2024-03-30T12:42:35.844 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-30T12:42:35.844 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630730 --name yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi --verbose --description rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-snaps-few-objects-overwrites} -- /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630730/orig.config.yaml 2024-03-30T12:42:35.887 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-30T12:42:35.979 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630730 branch: wip-ceph_test_rados-partial-reads description: rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-snaps-few-objects-overwrites} email: yweinste@redhat.com first_in_suite: false job_id: '7630730' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-ceph_test_rados-partial-reads ceph: conf: global: enable experimental unrecoverable data corrupting features: '*' mon client directed command retry: 5 mon election default strategy: 3 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: osd ms inject internal delays: 0.002 ms inject socket failures: 2500 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd max backfills: 2 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true create_rbd_pool: false flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force sha1: 5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc rados: wait_for_all_active_clean_pgs: true thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 disable_objectstore_tool_tests: true workunit: branch: wip-ceph_test_rados-partial-reads sha1: 5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 9805 sha1: 5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-ceph_test_rados-partial-reads suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc targets: smithi067.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHc368b/ld8CXeRgZD2wZKDabNI1eXfuXEMNsxKrM0ylBNhZkR2Ojg21ljXQq705FBt6xWiWXgxuXG2KNvfnxwY= smithi082.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOSfXZVLelftUITjlRAgf/G/9OTlf8HfMEZz+qIw9jTkkEqgB1v5m3QDgDoN4xfB9xrvptDpktppp2QW52tuIEI= tasks: - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: chance_test_min_size: 3 timeout: 1200 - rados: clients: - client.0 ec_pool: true erasure_code_use_overwrites: true objects: 50 op_weights: copy_from: 50 delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-30_00:41:04 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-30T12:42:35.979 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa; will attempt to use it 2024-03-30T12:42:35.979 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa/tasks 2024-03-30T12:42:35.980 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-30T12:42:35.981 INFO:teuthology.task.internal:Checking packages... 2024-03-30T12:42:36.013 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc' 2024-03-30T12:42:36.013 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-30T12:42:36.013 INFO:teuthology.packaging:ref: None 2024-03-30T12:42:36.013 INFO:teuthology.packaging:tag: None 2024-03-30T12:42:36.013 INFO:teuthology.packaging:branch: wip-ceph_test_rados-partial-reads 2024-03-30T12:42:36.013 INFO:teuthology.packaging:sha1: 5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc 2024-03-30T12:42:36.013 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-ceph_test_rados-partial-reads 2024-03-30T12:42:36.158 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1864-g5dc2ea30-1jammy 2024-03-30T12:42:36.159 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-30T12:42:36.168 INFO:teuthology.task.internal:no buildpackages task found 2024-03-30T12:42:36.168 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-30T12:42:36.183 INFO:teuthology.task.internal:Saving configuration 2024-03-30T12:42:36.199 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-30T12:42:36.207 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T12:42:36.228 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630730', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 12:36:52.783281', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKb/sdVFgK6HLH4MgxO0+z0bITwfhMW1bXCLm39Fx6SC'} 2024-03-30T12:42:36.246 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630730', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 12:36:52.781685', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDkVxJpWhEvWsiHy3m7JEIx57Mq5aIjsY1QwDEYohctB'} 2024-03-30T12:42:36.246 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-30T12:42:36.254 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-30T12:42:36.254 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-30T12:42:36.254 INFO:teuthology.run_tasks:Running task console_log... 2024-03-30T12:42:36.325 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6a127b43a0>, signals=[15]) 2024-03-30T12:42:36.326 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-30T12:42:36.333 INFO:teuthology.task.internal:Opening connections... 2024-03-30T12:42:36.333 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2024-03-30T12:42:36.335 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T12:42:36.421 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi082.front.sepia.ceph.com 2024-03-30T12:42:36.422 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T12:42:36.502 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-30T12:42:36.510 DEBUG:teuthology.orchestra.run.smithi067:> uname -m 2024-03-30T12:42:36.515 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2024-03-30T12:42:36.515 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2024-03-30T12:42:36.565 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-30T12:42:36.565 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="Ubuntu" 2024-03-30T12:42:36.565 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="22.04" 2024-03-30T12:42:36.565 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-30T12:42:36.565 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_CODENAME=jammy 2024-03-30T12:42:36.565 INFO:teuthology.orchestra.run.smithi067.stdout:ID=ubuntu 2024-03-30T12:42:36.565 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE=debian 2024-03-30T12:42:36.565 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-30T12:42:36.566 INFO:teuthology.orchestra.run.smithi067.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-30T12:42:36.566 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-30T12:42:36.566 INFO:teuthology.orchestra.run.smithi067.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-30T12:42:36.566 INFO:teuthology.orchestra.run.smithi067.stdout:UBUNTU_CODENAME=jammy 2024-03-30T12:42:36.567 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2024-03-30T12:42:36.591 DEBUG:teuthology.orchestra.run.smithi082:> uname -m 2024-03-30T12:42:36.596 INFO:teuthology.orchestra.run.smithi082.stdout:x86_64 2024-03-30T12:42:36.596 DEBUG:teuthology.orchestra.run.smithi082:> cat /etc/os-release 2024-03-30T12:42:36.643 INFO:teuthology.orchestra.run.smithi082.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-30T12:42:36.643 INFO:teuthology.orchestra.run.smithi082.stdout:NAME="Ubuntu" 2024-03-30T12:42:36.643 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION_ID="22.04" 2024-03-30T12:42:36.643 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-30T12:42:36.643 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION_CODENAME=jammy 2024-03-30T12:42:36.644 INFO:teuthology.orchestra.run.smithi082.stdout:ID=ubuntu 2024-03-30T12:42:36.644 INFO:teuthology.orchestra.run.smithi082.stdout:ID_LIKE=debian 2024-03-30T12:42:36.644 INFO:teuthology.orchestra.run.smithi082.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-30T12:42:36.644 INFO:teuthology.orchestra.run.smithi082.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-30T12:42:36.644 INFO:teuthology.orchestra.run.smithi082.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-30T12:42:36.644 INFO:teuthology.orchestra.run.smithi082.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-30T12:42:36.644 INFO:teuthology.orchestra.run.smithi082.stdout:UBUNTU_CODENAME=jammy 2024-03-30T12:42:36.644 INFO:teuthology.lock.ops:Updating smithi082.front.sepia.ceph.com on lock server 2024-03-30T12:42:36.666 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-30T12:42:36.676 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-30T12:42:36.683 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-30T12:42:36.683 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2024-03-30T12:42:36.686 DEBUG:teuthology.orchestra.run.smithi082:> test '!' -e /home/ubuntu/cephtest 2024-03-30T12:42:36.690 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-30T12:42:36.698 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-30T12:42:36.698 DEBUG:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2024-03-30T12:42:36.731 DEBUG:teuthology.orchestra.run.smithi082:> test -z $(ls -A /var/lib/ceph) 2024-03-30T12:42:36.744 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-30T12:42:36.797 INFO:teuthology.run_tasks:Running task kernel... 2024-03-30T12:42:36.815 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-30T12:42:36.815 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-30T12:42:36.815 DEBUG:teuthology.orchestra.run.smithi067:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-30T12:42:36.816 DEBUG:teuthology.orchestra.run.smithi082:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-30T12:42:36.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T12:42:36.821 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-03-30T12:42:36.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T12:42:36.822 DEBUG:teuthology.orchestra.run.smithi082:> uname -r 2024-03-30T12:42:36.868 INFO:teuthology.orchestra.run.smithi082.stdout:5.15.0-56-generic 2024-03-30T12:42:36.868 INFO:teuthology.task.kernel:Running kernel on smithi082: 5.15.0-56-generic 2024-03-30T12:42:36.868 DEBUG:teuthology.orchestra.run.smithi082:> sudo apt-get clean 2024-03-30T12:42:36.869 INFO:teuthology.orchestra.run.smithi067.stdout:5.15.0-56-generic 2024-03-30T12:42:36.869 INFO:teuthology.task.kernel:Running kernel on smithi067: 5.15.0-56-generic 2024-03-30T12:42:36.870 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get clean 2024-03-30T12:42:36.996 DEBUG:teuthology.orchestra.run.smithi082:> sudo apt-get update 2024-03-30T12:42:37.001 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get update 2024-03-30T12:42:37.142 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-30T12:42:37.142 INFO:teuthology.orchestra.run.smithi067.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-30T12:42:37.143 INFO:teuthology.orchestra.run.smithi082.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-30T12:42:37.147 INFO:teuthology.orchestra.run.smithi067.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-30T12:42:37.233 INFO:teuthology.orchestra.run.smithi067.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-03-30T12:42:37.245 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-30T12:42:37.328 INFO:teuthology.orchestra.run.smithi082.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-30T12:42:37.359 INFO:teuthology.orchestra.run.smithi082.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [436 kB] 2024-03-30T12:42:37.391 INFO:teuthology.orchestra.run.smithi067.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,303 kB] 2024-03-30T12:42:37.445 INFO:teuthology.orchestra.run.smithi082.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,303 kB] 2024-03-30T12:42:37.466 INFO:teuthology.orchestra.run.smithi067.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [602 kB] 2024-03-30T12:42:37.486 INFO:teuthology.orchestra.run.smithi082.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [233 kB] 2024-03-30T12:42:37.489 INFO:teuthology.orchestra.run.smithi082.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-03-30T12:42:37.489 INFO:teuthology.orchestra.run.smithi082.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-03-30T12:42:37.490 INFO:teuthology.orchestra.run.smithi082.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,616 kB] 2024-03-30T12:42:37.504 INFO:teuthology.orchestra.run.smithi067.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [436 kB] 2024-03-30T12:42:37.514 INFO:teuthology.orchestra.run.smithi067.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [233 kB] 2024-03-30T12:42:37.517 INFO:teuthology.orchestra.run.smithi082.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [271 kB] 2024-03-30T12:42:37.522 INFO:teuthology.orchestra.run.smithi082.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-03-30T12:42:37.522 INFO:teuthology.orchestra.run.smithi082.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [599 kB] 2024-03-30T12:42:37.525 INFO:teuthology.orchestra.run.smithi067.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-03-30T12:42:37.525 INFO:teuthology.orchestra.run.smithi067.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [33.7 kB] 2024-03-30T12:42:37.525 INFO:teuthology.orchestra.run.smithi067.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,616 kB] 2024-03-30T12:42:37.534 INFO:teuthology.orchestra.run.smithi082.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [852 kB] 2024-03-30T12:42:37.546 INFO:teuthology.orchestra.run.smithi082.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-03-30T12:42:37.548 INFO:teuthology.orchestra.run.smithi082.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-03-30T12:42:37.563 INFO:teuthology.orchestra.run.smithi067.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [271 kB] 2024-03-30T12:42:37.564 INFO:teuthology.orchestra.run.smithi082.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-03-30T12:42:37.565 INFO:teuthology.orchestra.run.smithi082.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-03-30T12:42:37.568 INFO:teuthology.orchestra.run.smithi067.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-03-30T12:42:37.568 INFO:teuthology.orchestra.run.smithi067.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [599 kB] 2024-03-30T12:42:37.576 INFO:teuthology.orchestra.run.smithi067.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,519 kB] 2024-03-30T12:42:37.580 INFO:teuthology.orchestra.run.smithi067.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [852 kB] 2024-03-30T12:42:37.587 INFO:teuthology.orchestra.run.smithi082.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-03-30T12:42:37.587 INFO:teuthology.orchestra.run.smithi082.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-03-30T12:42:37.598 INFO:teuthology.orchestra.run.smithi067.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-03-30T12:42:37.601 INFO:teuthology.orchestra.run.smithi067.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-03-30T12:42:37.625 INFO:teuthology.orchestra.run.smithi067.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-03-30T12:42:37.625 INFO:teuthology.orchestra.run.smithi067.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.1 kB] 2024-03-30T12:42:37.625 INFO:teuthology.orchestra.run.smithi067.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,476 B] 2024-03-30T12:42:37.631 INFO:teuthology.orchestra.run.smithi067.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-03-30T12:42:37.645 INFO:teuthology.orchestra.run.smithi067.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [293 kB] 2024-03-30T12:42:37.654 INFO:teuthology.orchestra.run.smithi067.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-03-30T12:42:37.655 INFO:teuthology.orchestra.run.smithi067.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,644 kB] 2024-03-30T12:42:37.681 INFO:teuthology.orchestra.run.smithi082.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-03-30T12:42:37.685 INFO:teuthology.orchestra.run.smithi067.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-03-30T12:42:37.686 INFO:teuthology.orchestra.run.smithi067.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [274 kB] 2024-03-30T12:42:37.690 INFO:teuthology.orchestra.run.smithi067.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-03-30T12:42:37.690 INFO:teuthology.orchestra.run.smithi067.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [698 kB] 2024-03-30T12:42:37.700 INFO:teuthology.orchestra.run.smithi067.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,060 kB] 2024-03-30T12:42:37.713 INFO:teuthology.orchestra.run.smithi067.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [241 kB] 2024-03-30T12:42:37.716 INFO:teuthology.orchestra.run.smithi067.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-03-30T12:42:37.716 INFO:teuthology.orchestra.run.smithi067.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [49.6 kB] 2024-03-30T12:42:37.717 INFO:teuthology.orchestra.run.smithi067.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [6,028 B] 2024-03-30T12:42:37.742 INFO:teuthology.orchestra.run.smithi067.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [12.0 kB] 2024-03-30T12:42:37.742 INFO:teuthology.orchestra.run.smithi067.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-03-30T12:42:37.742 INFO:teuthology.orchestra.run.smithi067.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-03-30T12:42:37.742 INFO:teuthology.orchestra.run.smithi067.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-03-30T12:42:37.742 INFO:teuthology.orchestra.run.smithi067.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-03-30T12:42:37.743 INFO:teuthology.orchestra.run.smithi067.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-03-30T12:42:37.761 INFO:teuthology.orchestra.run.smithi067.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-03-30T12:42:37.762 INFO:teuthology.orchestra.run.smithi067.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-03-30T12:42:37.762 INFO:teuthology.orchestra.run.smithi067.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-03-30T12:42:37.762 INFO:teuthology.orchestra.run.smithi067.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-03-30T12:42:37.820 INFO:teuthology.orchestra.run.smithi082.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [602 kB] 2024-03-30T12:42:38.017 INFO:teuthology.orchestra.run.smithi082.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,519 kB] 2024-03-30T12:42:38.165 INFO:teuthology.orchestra.run.smithi082.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [293 kB] 2024-03-30T12:42:38.179 INFO:teuthology.orchestra.run.smithi082.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-03-30T12:42:38.180 INFO:teuthology.orchestra.run.smithi082.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,644 kB] 2024-03-30T12:42:38.240 INFO:teuthology.orchestra.run.smithi082.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [35.3 kB] 2024-03-30T12:42:38.240 INFO:teuthology.orchestra.run.smithi082.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [274 kB] 2024-03-30T12:42:38.248 INFO:teuthology.orchestra.run.smithi082.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-03-30T12:42:38.412 INFO:teuthology.orchestra.run.smithi082.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,060 kB] 2024-03-30T12:42:39.050 INFO:teuthology.orchestra.run.smithi082.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [698 kB] 2024-03-30T12:42:39.088 INFO:teuthology.orchestra.run.smithi082.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [241 kB] 2024-03-30T12:42:39.101 INFO:teuthology.orchestra.run.smithi082.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-03-30T12:42:39.101 INFO:teuthology.orchestra.run.smithi082.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [6,028 B] 2024-03-30T12:42:39.102 INFO:teuthology.orchestra.run.smithi082.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [49.6 kB] 2024-03-30T12:42:39.105 INFO:teuthology.orchestra.run.smithi082.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [12.0 kB] 2024-03-30T12:42:39.105 INFO:teuthology.orchestra.run.smithi082.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-03-30T12:42:39.106 INFO:teuthology.orchestra.run.smithi082.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-03-30T12:42:39.132 INFO:teuthology.orchestra.run.smithi082.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-03-30T12:42:39.135 INFO:teuthology.orchestra.run.smithi082.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-03-30T12:42:39.300 INFO:teuthology.orchestra.run.smithi082.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-03-30T12:42:39.382 INFO:teuthology.orchestra.run.smithi082.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-03-30T12:42:39.484 INFO:teuthology.orchestra.run.smithi082.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-03-30T12:42:39.545 INFO:teuthology.orchestra.run.smithi082.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-03-30T12:42:39.575 INFO:teuthology.orchestra.run.smithi082.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-03-30T12:42:43.618 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 12.6 MB in 2s (6,787 kB/s) 2024-03-30T12:42:43.922 INFO:teuthology.orchestra.run.smithi082.stdout:Fetched 12.6 MB in 3s (5,015 kB/s) 2024-03-30T12:42:44.934 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-03-30T12:42:44.952 DEBUG:teuthology.orchestra.run.smithi067:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-30T12:42:45.013 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-03-30T12:42:45.213 INFO:teuthology.orchestra.run.smithi067.stdout:Building dependency tree... 2024-03-30T12:42:45.213 INFO:teuthology.orchestra.run.smithi067.stdout:Reading state information... 2024-03-30T12:42:45.389 INFO:teuthology.orchestra.run.smithi067.stdout:The following packages were automatically installed and are no longer required: 2024-03-30T12:42:45.389 INFO:teuthology.orchestra.run.smithi067.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-30T12:42:45.389 INFO:teuthology.orchestra.run.smithi067.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-30T12:42:45.390 INFO:teuthology.orchestra.run.smithi067.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-30T12:42:45.390 INFO:teuthology.orchestra.run.smithi067.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-03-30T12:42:45.390 INFO:teuthology.orchestra.run.smithi067.stdout: python2.7-minimal 2024-03-30T12:42:45.390 INFO:teuthology.orchestra.run.smithi067.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-30T12:42:45.391 INFO:teuthology.orchestra.run.smithi067.stdout:The following additional packages will be installed: 2024-03-30T12:42:45.391 INFO:teuthology.orchestra.run.smithi067.stdout: linux-generic linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-30T12:42:45.391 INFO:teuthology.orchestra.run.smithi067.stdout: linux-headers-generic linux-image-5.15.0-101-generic 2024-03-30T12:42:45.391 INFO:teuthology.orchestra.run.smithi067.stdout: linux-modules-5.15.0-101-generic linux-modules-extra-5.15.0-101-generic 2024-03-30T12:42:45.393 INFO:teuthology.orchestra.run.smithi067.stdout:Suggested packages: 2024-03-30T12:42:45.393 INFO:teuthology.orchestra.run.smithi067.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-03-30T12:42:45.393 INFO:teuthology.orchestra.run.smithi067.stdout:Recommended packages: 2024-03-30T12:42:45.393 INFO:teuthology.orchestra.run.smithi067.stdout: thermald 2024-03-30T12:42:45.428 INFO:teuthology.orchestra.run.smithi082.stdout:Reading package lists... 2024-03-30T12:42:45.434 INFO:teuthology.orchestra.run.smithi067.stdout:The following NEW packages will be installed: 2024-03-30T12:42:45.434 INFO:teuthology.orchestra.run.smithi067.stdout: linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-30T12:42:45.434 INFO:teuthology.orchestra.run.smithi067.stdout: linux-image-5.15.0-101-generic linux-modules-5.15.0-101-generic 2024-03-30T12:42:45.434 INFO:teuthology.orchestra.run.smithi067.stdout: linux-modules-extra-5.15.0-101-generic 2024-03-30T12:42:45.436 INFO:teuthology.orchestra.run.smithi067.stdout:The following packages will be upgraded: 2024-03-30T12:42:45.436 INFO:teuthology.orchestra.run.smithi067.stdout: linux-generic linux-headers-generic linux-image-generic 2024-03-30T12:42:45.446 DEBUG:teuthology.orchestra.run.smithi082:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-30T12:42:45.495 INFO:teuthology.orchestra.run.smithi067.stdout:3 upgraded, 5 newly installed, 0 to remove and 311 not upgraded. 2024-03-30T12:42:45.495 INFO:teuthology.orchestra.run.smithi067.stdout:Need to get 113 MB of archives. 2024-03-30T12:42:45.495 INFO:teuthology.orchestra.run.smithi067.stdout:After this operation, 582 MB of additional disk space will be used. 2024-03-30T12:42:45.495 INFO:teuthology.orchestra.run.smithi067.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-101-generic amd64 5.15.0-101.111 [22.6 MB] 2024-03-30T12:42:45.501 INFO:teuthology.orchestra.run.smithi082.stdout:Reading package lists... 2024-03-30T12:42:45.704 INFO:teuthology.orchestra.run.smithi082.stdout:Building dependency tree... 2024-03-30T12:42:45.704 INFO:teuthology.orchestra.run.smithi082.stdout:Reading state information... 2024-03-30T12:42:45.866 INFO:teuthology.orchestra.run.smithi082.stdout:The following packages were automatically installed and are no longer required: 2024-03-30T12:42:45.866 INFO:teuthology.orchestra.run.smithi082.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-30T12:42:45.866 INFO:teuthology.orchestra.run.smithi082.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-30T12:42:45.867 INFO:teuthology.orchestra.run.smithi082.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-30T12:42:45.868 INFO:teuthology.orchestra.run.smithi082.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-03-30T12:42:45.868 INFO:teuthology.orchestra.run.smithi082.stdout: python2.7-minimal 2024-03-30T12:42:45.868 INFO:teuthology.orchestra.run.smithi082.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-30T12:42:45.868 INFO:teuthology.orchestra.run.smithi082.stdout:The following additional packages will be installed: 2024-03-30T12:42:45.868 INFO:teuthology.orchestra.run.smithi082.stdout: linux-generic linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-30T12:42:45.868 INFO:teuthology.orchestra.run.smithi082.stdout: linux-headers-generic linux-image-5.15.0-101-generic 2024-03-30T12:42:45.868 INFO:teuthology.orchestra.run.smithi082.stdout: linux-modules-5.15.0-101-generic linux-modules-extra-5.15.0-101-generic 2024-03-30T12:42:45.869 INFO:teuthology.orchestra.run.smithi082.stdout:Suggested packages: 2024-03-30T12:42:45.870 INFO:teuthology.orchestra.run.smithi082.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-03-30T12:42:45.870 INFO:teuthology.orchestra.run.smithi082.stdout:Recommended packages: 2024-03-30T12:42:45.870 INFO:teuthology.orchestra.run.smithi082.stdout: thermald 2024-03-30T12:42:45.916 INFO:teuthology.orchestra.run.smithi082.stdout:The following NEW packages will be installed: 2024-03-30T12:42:45.916 INFO:teuthology.orchestra.run.smithi082.stdout: linux-headers-5.15.0-101 linux-headers-5.15.0-101-generic 2024-03-30T12:42:45.916 INFO:teuthology.orchestra.run.smithi082.stdout: linux-image-5.15.0-101-generic linux-modules-5.15.0-101-generic 2024-03-30T12:42:45.916 INFO:teuthology.orchestra.run.smithi082.stdout: linux-modules-extra-5.15.0-101-generic 2024-03-30T12:42:45.917 INFO:teuthology.orchestra.run.smithi082.stdout:The following packages will be upgraded: 2024-03-30T12:42:45.917 INFO:teuthology.orchestra.run.smithi082.stdout: linux-generic linux-headers-generic linux-image-generic 2024-03-30T12:42:45.973 INFO:teuthology.orchestra.run.smithi082.stdout:3 upgraded, 5 newly installed, 0 to remove and 311 not upgraded. 2024-03-30T12:42:45.974 INFO:teuthology.orchestra.run.smithi082.stdout:Need to get 113 MB of archives. 2024-03-30T12:42:45.974 INFO:teuthology.orchestra.run.smithi082.stdout:After this operation, 582 MB of additional disk space will be used. 2024-03-30T12:42:45.974 INFO:teuthology.orchestra.run.smithi082.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-101-generic amd64 5.15.0-101.111 [22.6 MB] 2024-03-30T12:42:46.014 INFO:teuthology.orchestra.run.smithi067.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-101-generic amd64 5.15.0-101.111 [11.5 MB] 2024-03-30T12:42:46.165 INFO:teuthology.orchestra.run.smithi067.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-101-generic amd64 5.15.0-101.111 [63.9 MB] 2024-03-30T12:42:46.876 INFO:teuthology.orchestra.run.smithi082.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-101-generic amd64 5.15.0-101.111 [11.5 MB] 2024-03-30T12:42:47.244 INFO:teuthology.orchestra.run.smithi082.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-101-generic amd64 5.15.0-101.111 [63.9 MB] 2024-03-30T12:42:47.887 INFO:teuthology.orchestra.run.smithi067.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.101.98 [1,698 B] 2024-03-30T12:42:47.887 INFO:teuthology.orchestra.run.smithi067.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.101.98 [2,486 B] 2024-03-30T12:42:47.887 INFO:teuthology.orchestra.run.smithi067.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101 all 5.15.0-101.111 [12.3 MB] 2024-03-30T12:42:48.127 INFO:teuthology.orchestra.run.smithi067.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101-generic amd64 5.15.0-101.111 [2,876 kB] 2024-03-30T12:42:48.251 INFO:teuthology.orchestra.run.smithi067.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.101.98 [2,332 B] 2024-03-30T12:42:48.651 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 113 MB in 3s (40.3 MB/s) 2024-03-30T12:42:48.674 INFO:teuthology.orchestra.run.smithi082.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.101.98 [1,698 B] 2024-03-30T12:42:48.674 INFO:teuthology.orchestra.run.smithi082.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.101.98 [2,486 B] 2024-03-30T12:42:48.675 INFO:teuthology.orchestra.run.smithi082.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101 all 5.15.0-101.111 [12.3 MB] 2024-03-30T12:42:48.817 INFO:teuthology.orchestra.run.smithi082.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-101-generic amd64 5.15.0-101.111 [2,876 kB] 2024-03-30T12:42:48.840 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-modules-5.15.0-101-generic. 2024-03-30T12:42:48.909 INFO:teuthology.orchestra.run.smithi082.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.101.98 [2,332 B] 2024-03-30T12:42:49.306 INFO:teuthology.orchestra.run.smithi082.stdout:Fetched 113 MB in 3s (37.9 MB/s) 2024-03-30T12:42:49.475 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package linux-modules-5.15.0-101-generic. 2024-03-30T12:42:50.860 INFO:teuthology.orchestra.run.smithi067.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-03-30T12:42:50.865 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../0-linux-modules-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-30T12:42:51.170 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T12:42:51.493 INFO:teuthology.orchestra.run.smithi082.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-03-30T12:42:51.498 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../0-linux-modules-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-30T12:42:51.639 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T12:42:53.480 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-image-5.15.0-101-generic. 2024-03-30T12:42:53.501 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../1-linux-image-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-30T12:42:53.598 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T12:42:54.007 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package linux-image-5.15.0-101-generic. 2024-03-30T12:42:54.028 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../1-linux-image-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-30T12:42:54.121 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T12:42:54.200 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-101-generic. 2024-03-30T12:42:54.221 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-30T12:42:54.258 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T12:42:54.585 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-101-generic. 2024-03-30T12:42:54.606 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-30T12:42:54.635 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T12:43:00.835 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../3-linux-generic_5.15.0.101.98_amd64.deb ... 2024-03-30T12:43:00.953 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-30T12:43:01.311 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.101.98_amd64.deb ... 2024-03-30T12:43:01.440 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-image-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-30T12:43:01.737 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../3-linux-generic_5.15.0.101.98_amd64.deb ... 2024-03-30T12:43:01.742 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-headers-5.15.0-101. 2024-03-30T12:43:01.765 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../5-linux-headers-5.15.0-101_5.15.0-101.111_all.deb ... 2024-03-30T12:43:01.809 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-30T12:43:01.871 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-30T12:43:02.188 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.101.98_amd64.deb ... 2024-03-30T12:43:02.333 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-image-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-30T12:43:02.636 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package linux-headers-5.15.0-101. 2024-03-30T12:43:02.658 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../5-linux-headers-5.15.0-101_5.15.0-101.111_all.deb ... 2024-03-30T12:43:02.702 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-30T12:43:07.324 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-headers-5.15.0-101-generic. 2024-03-30T12:43:07.361 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../6-linux-headers-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-30T12:43:07.407 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T12:43:08.596 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package linux-headers-5.15.0-101-generic. 2024-03-30T12:43:08.624 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../6-linux-headers-5.15.0-101-generic_5.15.0-101.111_amd64.deb ... 2024-03-30T12:43:08.683 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T12:43:10.164 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.101.98_amd64.deb ... 2024-03-30T12:43:10.303 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-headers-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-30T12:43:10.653 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-30T12:43:10.799 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T12:43:10.925 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-headers-generic (5.15.0.101.98) ... 2024-03-30T12:43:11.054 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T12:43:12.071 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.101.98_amd64.deb ... 2024-03-30T12:43:12.205 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking linux-headers-generic (5.15.0.101.98) over (5.15.0.56.54) ... 2024-03-30T12:43:12.521 INFO:teuthology.orchestra.run.smithi067.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-03-30T12:43:12.521 INFO:teuthology.orchestra.run.smithi067.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-03-30T12:43:12.521 INFO:teuthology.orchestra.run.smithi067.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-101-generic 2024-03-30T12:43:12.521 INFO:teuthology.orchestra.run.smithi067.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-101-generic 2024-03-30T12:43:12.554 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-headers-5.15.0-101 (5.15.0-101.111) ... 2024-03-30T12:43:12.684 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-headers-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T12:43:13.511 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-headers-generic (5.15.0.101.98) ... 2024-03-30T12:43:13.640 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T12:43:13.732 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T12:43:14.984 INFO:teuthology.orchestra.run.smithi082.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-03-30T12:43:14.984 INFO:teuthology.orchestra.run.smithi082.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-03-30T12:43:14.984 INFO:teuthology.orchestra.run.smithi082.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-101-generic 2024-03-30T12:43:14.984 INFO:teuthology.orchestra.run.smithi082.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-101-generic 2024-03-30T12:43:15.142 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-modules-extra-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T12:43:15.157 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T12:43:16.550 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-image-generic (5.15.0.101.98) ... 2024-03-30T12:43:16.567 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-modules-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T12:43:16.676 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-generic (5.15.0.101.98) ... 2024-03-30T12:43:16.803 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T12:43:16.875 INFO:teuthology.orchestra.run.smithi067.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-03-30T12:43:16.875 INFO:teuthology.orchestra.run.smithi067.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-101-generic 2024-03-30T12:43:17.968 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-image-generic (5.15.0.101.98) ... 2024-03-30T12:43:18.111 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up linux-generic (5.15.0.101.98) ... 2024-03-30T12:43:18.237 INFO:teuthology.orchestra.run.smithi082.stdout:Processing triggers for linux-image-5.15.0-101-generic (5.15.0-101.111) ... 2024-03-30T12:43:18.294 INFO:teuthology.orchestra.run.smithi082.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-03-30T12:43:18.294 INFO:teuthology.orchestra.run.smithi082.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-101-generic 2024-03-30T12:43:31.247 INFO:teuthology.orchestra.run.smithi067.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-03-30T12:43:31.247 INFO:teuthology.orchestra.run.smithi067.stdout:Sourcing file `/etc/default/grub' 2024-03-30T12:43:31.279 INFO:teuthology.orchestra.run.smithi067.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-30T12:43:31.310 INFO:teuthology.orchestra.run.smithi067.stdout:Generating grub configuration file ... 2024-03-30T12:43:31.557 INFO:teuthology.orchestra.run.smithi067.stdout:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-30T12:43:31.577 INFO:teuthology.orchestra.run.smithi067.stdout:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-30T12:43:31.878 INFO:teuthology.orchestra.run.smithi067.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-30T12:43:31.883 INFO:teuthology.orchestra.run.smithi067.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-30T12:43:31.957 INFO:teuthology.orchestra.run.smithi067.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-30T12:43:31.963 INFO:teuthology.orchestra.run.smithi067.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-30T12:43:32.144 INFO:teuthology.orchestra.run.smithi067.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-30T12:43:32.144 INFO:teuthology.orchestra.run.smithi067.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-03-30T12:43:32.144 INFO:teuthology.orchestra.run.smithi067.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-30T12:43:32.184 INFO:teuthology.orchestra.run.smithi067.stdout:done 2024-03-30T12:43:32.608 INFO:teuthology.orchestra.run.smithi082.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-03-30T12:43:32.608 INFO:teuthology.orchestra.run.smithi082.stdout:Sourcing file `/etc/default/grub' 2024-03-30T12:43:32.622 DEBUG:teuthology.orchestra.run.smithi067:> dpkg -s linux-image-generic 2024-03-30T12:43:32.633 INFO:teuthology.orchestra.run.smithi082.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-30T12:43:32.642 INFO:teuthology.orchestra.run.smithi067.stdout:Package: linux-image-generic 2024-03-30T12:43:32.642 INFO:teuthology.orchestra.run.smithi067.stdout:Status: install ok installed 2024-03-30T12:43:32.642 INFO:teuthology.orchestra.run.smithi067.stdout:Priority: optional 2024-03-30T12:43:32.642 INFO:teuthology.orchestra.run.smithi067.stdout:Section: kernel 2024-03-30T12:43:32.642 INFO:teuthology.orchestra.run.smithi067.stdout:Installed-Size: 21 2024-03-30T12:43:32.642 INFO:teuthology.orchestra.run.smithi067.stdout:Maintainer: Ubuntu Kernel Team 2024-03-30T12:43:32.642 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: amd64 2024-03-30T12:43:32.642 INFO:teuthology.orchestra.run.smithi067.stdout:Source: linux-meta 2024-03-30T12:43:32.642 INFO:teuthology.orchestra.run.smithi067.stdout:Version: 5.15.0.101.98 2024-03-30T12:43:32.643 INFO:teuthology.orchestra.run.smithi067.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-30T12:43:32.643 INFO:teuthology.orchestra.run.smithi067.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-30T12:43:32.643 INFO:teuthology.orchestra.run.smithi067.stdout:Recommends: thermald 2024-03-30T12:43:32.643 INFO:teuthology.orchestra.run.smithi067.stdout:Description: Generic Linux kernel image 2024-03-30T12:43:32.643 INFO:teuthology.orchestra.run.smithi067.stdout: This package will always depend on the latest generic kernel image 2024-03-30T12:43:32.643 INFO:teuthology.orchestra.run.smithi067.stdout: available. 2024-03-30T12:43:32.643 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-101-generic 2024-03-30T12:43:32.643 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-30T12:43:32.643 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-30T12:43:32.644 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-30T12:43:32.644 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi067.front.sepia.ceph.com, path=None, version=distro) 2024-03-30T12:43:32.644 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get clean 2024-03-30T12:43:32.656 INFO:teuthology.orchestra.run.smithi082.stdout:Generating grub configuration file ... 2024-03-30T12:43:32.824 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get update 2024-03-30T12:43:32.952 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-30T12:43:32.953 INFO:teuthology.orchestra.run.smithi067.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-30T12:43:32.965 INFO:teuthology.orchestra.run.smithi067.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-03-30T12:43:32.998 INFO:teuthology.orchestra.run.smithi082.stdout:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-30T12:43:33.028 INFO:teuthology.orchestra.run.smithi082.stdout:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-30T12:43:33.051 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-30T12:43:33.378 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 229 kB in 0s (592 kB/s) 2024-03-30T12:43:33.379 INFO:teuthology.orchestra.run.smithi082.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-30T12:43:33.385 INFO:teuthology.orchestra.run.smithi082.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-30T12:43:33.460 INFO:teuthology.orchestra.run.smithi082.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-30T12:43:33.466 INFO:teuthology.orchestra.run.smithi082.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-30T12:43:33.637 INFO:teuthology.orchestra.run.smithi082.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-30T12:43:33.637 INFO:teuthology.orchestra.run.smithi082.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-03-30T12:43:33.637 INFO:teuthology.orchestra.run.smithi082.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-30T12:43:33.685 INFO:teuthology.orchestra.run.smithi082.stdout:done 2024-03-30T12:43:34.186 DEBUG:teuthology.orchestra.run.smithi082:> dpkg -s linux-image-generic 2024-03-30T12:43:34.205 INFO:teuthology.orchestra.run.smithi082.stdout:Package: linux-image-generic 2024-03-30T12:43:34.205 INFO:teuthology.orchestra.run.smithi082.stdout:Status: install ok installed 2024-03-30T12:43:34.205 INFO:teuthology.orchestra.run.smithi082.stdout:Priority: optional 2024-03-30T12:43:34.206 INFO:teuthology.orchestra.run.smithi082.stdout:Section: kernel 2024-03-30T12:43:34.206 INFO:teuthology.orchestra.run.smithi082.stdout:Installed-Size: 21 2024-03-30T12:43:34.206 INFO:teuthology.orchestra.run.smithi082.stdout:Maintainer: Ubuntu Kernel Team 2024-03-30T12:43:34.206 INFO:teuthology.orchestra.run.smithi082.stdout:Architecture: amd64 2024-03-30T12:43:34.206 INFO:teuthology.orchestra.run.smithi082.stdout:Source: linux-meta 2024-03-30T12:43:34.206 INFO:teuthology.orchestra.run.smithi082.stdout:Version: 5.15.0.101.98 2024-03-30T12:43:34.206 INFO:teuthology.orchestra.run.smithi082.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-30T12:43:34.206 INFO:teuthology.orchestra.run.smithi082.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-30T12:43:34.206 INFO:teuthology.orchestra.run.smithi082.stdout:Recommends: thermald 2024-03-30T12:43:34.206 INFO:teuthology.orchestra.run.smithi082.stdout:Description: Generic Linux kernel image 2024-03-30T12:43:34.206 INFO:teuthology.orchestra.run.smithi082.stdout: This package will always depend on the latest generic kernel image 2024-03-30T12:43:34.206 INFO:teuthology.orchestra.run.smithi082.stdout: available. 2024-03-30T12:43:34.207 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-101-generic 2024-03-30T12:43:34.207 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-30T12:43:34.207 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-30T12:43:34.207 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-30T12:43:34.207 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi082.front.sepia.ceph.com, path=None, version=distro) 2024-03-30T12:43:34.207 DEBUG:teuthology.orchestra.run.smithi082:> sudo apt-get clean 2024-03-30T12:43:34.312 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-03-30T12:43:34.329 DEBUG:teuthology.orchestra.run.smithi067:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-30T12:43:34.382 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-03-30T12:43:34.386 DEBUG:teuthology.orchestra.run.smithi082:> sudo apt-get update 2024-03-30T12:43:34.503 INFO:teuthology.orchestra.run.smithi082.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-03-30T12:43:34.515 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-30T12:43:34.527 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-30T12:43:34.547 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-30T12:43:34.560 INFO:teuthology.orchestra.run.smithi067.stdout:Building dependency tree... 2024-03-30T12:43:34.561 INFO:teuthology.orchestra.run.smithi067.stdout:Reading state information... 2024-03-30T12:43:34.721 INFO:teuthology.orchestra.run.smithi067.stdout:linux-image-generic is already the newest version (5.15.0.101.98). 2024-03-30T12:43:34.722 INFO:teuthology.orchestra.run.smithi067.stdout:The following packages were automatically installed and are no longer required: 2024-03-30T12:43:34.722 INFO:teuthology.orchestra.run.smithi067.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-30T12:43:34.722 INFO:teuthology.orchestra.run.smithi067.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-30T12:43:34.722 INFO:teuthology.orchestra.run.smithi067.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-30T12:43:34.722 INFO:teuthology.orchestra.run.smithi067.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-30T12:43:34.722 INFO:teuthology.orchestra.run.smithi067.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-30T12:43:34.722 INFO:teuthology.orchestra.run.smithi067.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-30T12:43:34.762 INFO:teuthology.orchestra.run.smithi067.stdout:0 upgraded, 0 newly installed, 0 to remove and 311 not upgraded. 2024-03-30T12:43:34.764 DEBUG:teuthology.orchestra.run.smithi067:> dpkg -s linux-image-generic 2024-03-30T12:43:34.821 INFO:teuthology.orchestra.run.smithi067.stdout:Package: linux-image-generic 2024-03-30T12:43:34.822 INFO:teuthology.orchestra.run.smithi067.stdout:Status: install ok installed 2024-03-30T12:43:34.822 INFO:teuthology.orchestra.run.smithi067.stdout:Priority: optional 2024-03-30T12:43:34.822 INFO:teuthology.orchestra.run.smithi067.stdout:Section: kernel 2024-03-30T12:43:34.822 INFO:teuthology.orchestra.run.smithi067.stdout:Installed-Size: 21 2024-03-30T12:43:34.822 INFO:teuthology.orchestra.run.smithi067.stdout:Maintainer: Ubuntu Kernel Team 2024-03-30T12:43:34.822 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: amd64 2024-03-30T12:43:34.822 INFO:teuthology.orchestra.run.smithi067.stdout:Source: linux-meta 2024-03-30T12:43:34.822 INFO:teuthology.orchestra.run.smithi067.stdout:Version: 5.15.0.101.98 2024-03-30T12:43:34.822 INFO:teuthology.orchestra.run.smithi067.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-30T12:43:34.823 INFO:teuthology.orchestra.run.smithi067.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-30T12:43:34.823 INFO:teuthology.orchestra.run.smithi067.stdout:Recommends: thermald 2024-03-30T12:43:34.823 INFO:teuthology.orchestra.run.smithi067.stdout:Description: Generic Linux kernel image 2024-03-30T12:43:34.823 INFO:teuthology.orchestra.run.smithi067.stdout: This package will always depend on the latest generic kernel image 2024-03-30T12:43:34.823 INFO:teuthology.orchestra.run.smithi067.stdout: available. 2024-03-30T12:43:34.823 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-03-30T12:43:34.870 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.Wdfzl9FSgJ 2024-03-30T12:43:34.870 DEBUG:teuthology.orchestra.run.smithi067:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Wdfzl9FSgJ 2024-03-30T12:43:34.905 INFO:teuthology.orchestra.run.smithi082.stdout:Fetched 110 kB in 0s (296 kB/s) 2024-03-30T12:43:34.926 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.Wdfzl9FSgJ 2024-03-30T12:43:35.046 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.Wdfzl9FSgJ is 10KB 2024-03-30T12:43:35.060 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.Wdfzl9FSgJ 2024-03-30T12:43:35.066 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-03-30T12:43:35.122 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-30T12:43:35.122 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-03-30T12:43:35.122 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-03-30T12:43:35.190 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-101-generic 2024-03-30T12:43:35.190 DEBUG:teuthology.orchestra.run.smithi067:> sudo update-grub 2024-03-30T12:43:35.848 INFO:teuthology.orchestra.run.smithi082.stdout:Reading package lists... 2024-03-30T12:43:35.866 DEBUG:teuthology.orchestra.run.smithi082:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-03-30T12:43:35.925 INFO:teuthology.orchestra.run.smithi082.stdout:Reading package lists... 2024-03-30T12:43:36.122 INFO:teuthology.orchestra.run.smithi082.stdout:Building dependency tree... 2024-03-30T12:43:36.122 INFO:teuthology.orchestra.run.smithi082.stdout:Reading state information... 2024-03-30T12:43:36.284 INFO:teuthology.orchestra.run.smithi082.stdout:linux-image-generic is already the newest version (5.15.0.101.98). 2024-03-30T12:43:36.284 INFO:teuthology.orchestra.run.smithi082.stdout:The following packages were automatically installed and are no longer required: 2024-03-30T12:43:36.285 INFO:teuthology.orchestra.run.smithi082.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-30T12:43:36.285 INFO:teuthology.orchestra.run.smithi082.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-30T12:43:36.285 INFO:teuthology.orchestra.run.smithi082.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-30T12:43:36.285 INFO:teuthology.orchestra.run.smithi082.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-30T12:43:36.285 INFO:teuthology.orchestra.run.smithi082.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-30T12:43:36.285 INFO:teuthology.orchestra.run.smithi082.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-30T12:43:36.325 INFO:teuthology.orchestra.run.smithi082.stdout:0 upgraded, 0 newly installed, 0 to remove and 311 not upgraded. 2024-03-30T12:43:36.326 DEBUG:teuthology.orchestra.run.smithi082:> dpkg -s linux-image-generic 2024-03-30T12:43:36.342 INFO:teuthology.orchestra.run.smithi082.stdout:Package: linux-image-generic 2024-03-30T12:43:36.342 INFO:teuthology.orchestra.run.smithi082.stdout:Status: install ok installed 2024-03-30T12:43:36.342 INFO:teuthology.orchestra.run.smithi082.stdout:Priority: optional 2024-03-30T12:43:36.342 INFO:teuthology.orchestra.run.smithi082.stdout:Section: kernel 2024-03-30T12:43:36.342 INFO:teuthology.orchestra.run.smithi082.stdout:Installed-Size: 21 2024-03-30T12:43:36.342 INFO:teuthology.orchestra.run.smithi082.stdout:Maintainer: Ubuntu Kernel Team 2024-03-30T12:43:36.342 INFO:teuthology.orchestra.run.smithi082.stdout:Architecture: amd64 2024-03-30T12:43:36.342 INFO:teuthology.orchestra.run.smithi082.stdout:Source: linux-meta 2024-03-30T12:43:36.342 INFO:teuthology.orchestra.run.smithi082.stdout:Version: 5.15.0.101.98 2024-03-30T12:43:36.342 INFO:teuthology.orchestra.run.smithi082.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-101), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-03-30T12:43:36.342 INFO:teuthology.orchestra.run.smithi082.stdout:Depends: linux-image-5.15.0-101-generic, linux-modules-extra-5.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2024-03-30T12:43:36.342 INFO:teuthology.orchestra.run.smithi082.stdout:Recommends: thermald 2024-03-30T12:43:36.343 INFO:teuthology.orchestra.run.smithi082.stdout:Description: Generic Linux kernel image 2024-03-30T12:43:36.343 INFO:teuthology.orchestra.run.smithi082.stdout: This package will always depend on the latest generic kernel image 2024-03-30T12:43:36.343 INFO:teuthology.orchestra.run.smithi082.stdout: available. 2024-03-30T12:43:36.343 DEBUG:teuthology.orchestra.run.smithi082:> mktemp 2024-03-30T12:43:36.388 INFO:teuthology.orchestra.run.smithi082.stdout:/tmp/tmp.O0wgjfYNHe 2024-03-30T12:43:36.389 DEBUG:teuthology.orchestra.run.smithi082:> sudo cp /boot/grub/grub.cfg /tmp/tmp.O0wgjfYNHe 2024-03-30T12:43:36.436 INFO:teuthology.orchestra.run.smithi067.stderr:Sourcing file `/etc/default/grub' 2024-03-30T12:43:36.438 INFO:teuthology.orchestra.run.smithi067.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-30T12:43:36.440 INFO:teuthology.orchestra.run.smithi067.stderr:Generating grub configuration file ... 2024-03-30T12:43:36.444 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0666 /tmp/tmp.O0wgjfYNHe 2024-03-30T12:43:36.559 DEBUG:teuthology.orchestra.remote:smithi082:/tmp/tmp.O0wgjfYNHe is 10KB 2024-03-30T12:43:36.609 DEBUG:teuthology.orchestra.run.smithi082:> rm -fr /tmp/tmp.O0wgjfYNHe 2024-03-30T12:43:36.614 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-03-30T12:43:36.631 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-30T12:43:36.637 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-30T12:43:36.667 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-30T12:43:36.668 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-03-30T12:43:36.668 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-03-30T12:43:36.755 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-101-generic 2024-03-30T12:43:36.755 DEBUG:teuthology.orchestra.run.smithi082:> sudo update-grub 2024-03-30T12:43:36.961 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-30T12:43:36.965 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-30T12:43:37.004 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-30T12:43:37.009 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-30T12:43:37.147 INFO:teuthology.orchestra.run.smithi067.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-30T12:43:37.147 INFO:teuthology.orchestra.run.smithi067.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-30T12:43:37.147 INFO:teuthology.orchestra.run.smithi067.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-30T12:43:37.162 INFO:teuthology.orchestra.run.smithi067.stderr:done 2024-03-30T12:43:37.180 DEBUG:teuthology.orchestra.run.smithi067:> sudo shutdown -r now 2024-03-30T12:43:38.171 INFO:teuthology.orchestra.run.smithi082.stderr:Sourcing file `/etc/default/grub' 2024-03-30T12:43:38.172 INFO:teuthology.orchestra.run.smithi082.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-30T12:43:38.175 INFO:teuthology.orchestra.run.smithi082.stderr:Generating grub configuration file ... 2024-03-30T12:43:38.359 INFO:teuthology.orchestra.run.smithi082.stderr:Found linux image: /boot/vmlinuz-5.15.0-101-generic 2024-03-30T12:43:38.365 INFO:teuthology.orchestra.run.smithi082.stderr:Found initrd image: /boot/initrd.img-5.15.0-101-generic 2024-03-30T12:43:38.582 INFO:teuthology.orchestra.run.smithi082.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-30T12:43:38.587 INFO:teuthology.orchestra.run.smithi082.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-30T12:43:38.632 INFO:teuthology.orchestra.run.smithi082.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-30T12:43:38.637 INFO:teuthology.orchestra.run.smithi082.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-30T12:43:38.755 INFO:teuthology.orchestra.run.smithi082.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-30T12:43:38.755 INFO:teuthology.orchestra.run.smithi082.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-30T12:43:38.755 INFO:teuthology.orchestra.run.smithi082.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-30T12:43:38.790 INFO:teuthology.orchestra.run.smithi082.stderr:done 2024-03-30T12:43:38.798 DEBUG:teuthology.orchestra.run.smithi082:> sudo shutdown -r now 2024-03-30T12:44:07.210 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-30T12:44:07.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-30T12:44:07.212 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T12:44:08.802 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-30T12:44:08.803 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-03-30T12:44:08.803 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T12:44:25.762 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-03-30T12:44:27.294 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.82 2024-03-30T12:44:34.770 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-30T12:44:34.771 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T12:44:36.296 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-03-30T12:44:36.297 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T12:44:37.825 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-03-30T12:44:39.362 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.82 2024-03-30T12:44:49.836 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-30T12:44:49.838 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T12:44:51.365 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-03-30T12:44:51.366 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T12:45:08.258 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-03-30T12:45:09.794 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.82 2024-03-30T12:45:23.272 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-30T12:45:23.273 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T12:45:23.654 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-03-30T12:45:24.512 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-30T12:45:24.512 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-101-generic"... 2024-03-30T12:45:24.512 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-03-30T12:45:24.560 INFO:teuthology.orchestra.run.smithi067.stdout:5.15.0-101-generic 2024-03-30T12:45:24.560 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-101-generic vs 5.15.0-101-generic 2024-03-30T12:45:24.560 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-30T12:45:24.561 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-30T12:45:24.796 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-03-30T12:45:24.796 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T12:45:25.562 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-30T12:45:25.562 DEBUG:teuthology.orchestra.run.smithi067:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-30T12:45:25.648 INFO:teuthology.orchestra.run.smithi067.stdout:ttyS1 2024-03-30T12:45:25.669 DEBUG:teuthology.parallel:result is None 2024-03-30T12:45:25.687 DEBUG:teuthology.orchestra.run.smithi082:> true 2024-03-30T12:45:26.530 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-03-30T12:45:26.530 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-101-generic"... 2024-03-30T12:45:26.530 DEBUG:teuthology.orchestra.run.smithi082:> uname -r 2024-03-30T12:45:26.575 INFO:teuthology.orchestra.run.smithi082.stdout:5.15.0-101-generic 2024-03-30T12:45:26.576 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-101-generic vs 5.15.0-101-generic 2024-03-30T12:45:26.576 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-30T12:45:26.576 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-30T12:45:27.577 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-30T12:45:27.578 DEBUG:teuthology.orchestra.run.smithi082:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-30T12:45:27.705 INFO:teuthology.orchestra.run.smithi082.stdout:ttyS1 2024-03-30T12:45:27.729 DEBUG:teuthology.parallel:result is None 2024-03-30T12:45:27.729 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-30T12:45:27.743 INFO:teuthology.task.internal:Creating test directory... 2024-03-30T12:45:27.744 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-30T12:45:27.746 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-30T12:45:27.751 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-30T12:45:27.765 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-30T12:45:27.774 INFO:teuthology.task.internal:Creating archive directory... 2024-03-30T12:45:27.775 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-30T12:45:27.792 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-30T12:45:27.810 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-30T12:45:27.816 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-30T12:45:27.817 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-30T12:45:27.844 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-30T12:45:27.865 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:45:27.865 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:45:27.872 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:45:27.873 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-30T12:45:27.875 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-30T12:45:27.882 INFO:teuthology.task.internal:Configuring sudo... 2024-03-30T12:45:27.882 DEBUG:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-30T12:45:27.920 DEBUG:teuthology.orchestra.run.smithi082:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-30T12:45:27.943 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-30T12:45:27.951 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-30T12:45:27.951 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-30T12:45:27.976 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-30T12:45:27.990 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T12:45:28.027 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T12:45:28.075 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-30T12:45:28.075 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-30T12:45:28.142 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-30T12:45:28.148 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-30T12:45:28.196 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-30T12:45:28.196 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-30T12:45:28.280 DEBUG:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2024-03-30T12:45:28.283 DEBUG:teuthology.orchestra.run.smithi082:> sudo service rsyslog restart 2024-03-30T12:45:28.360 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-30T12:45:28.368 INFO:teuthology.task.internal:Starting timer... 2024-03-30T12:45:28.368 INFO:teuthology.run_tasks:Running task pcp... 2024-03-30T12:45:28.382 INFO:teuthology.run_tasks:Running task selinux... 2024-03-30T12:45:28.391 DEBUG:teuthology.task.selinux:Excluding smithi067: OS 'ubuntu' does not support SELinux 2024-03-30T12:45:28.391 DEBUG:teuthology.task.selinux:Excluding smithi082: OS 'ubuntu' does not support SELinux 2024-03-30T12:45:28.392 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-30T12:45:28.392 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-03-30T12:45:28.392 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-30T12:45:28.392 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-30T12:45:28.400 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-30T12:45:28.405 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-30T12:45:28.559 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-30T12:45:28.575 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-30T12:45:28.577 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi067.front.sepia.ceph.com,smithi082.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-30T12:52:34.448 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi067.front.sepia.ceph.com'), Remote(name='ubuntu@smithi082.front.sepia.ceph.com')] 2024-03-30T12:52:34.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-30T12:52:34.451 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T12:52:34.534 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-03-30T12:52:34.614 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-03-30T12:52:34.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-03-30T12:52:34.615 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-30T12:52:34.694 DEBUG:teuthology.orchestra.run.smithi082:> true 2024-03-30T12:52:34.770 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-03-30T12:52:34.770 INFO:teuthology.run_tasks:Running task clock... 2024-03-30T12:52:34.781 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-30T12:52:34.781 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-30T12:52:34.781 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T12:52:34.784 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-30T12:52:34.784 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T12:52:34.814 INFO:teuthology.orchestra.run.smithi067.stdout:30 Mar 12:52:34 ntpd[17196]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-30T12:52:34.814 INFO:teuthology.orchestra.run.smithi067.stdout:30 Mar 12:52:34 ntpd[17196]: Command line: ntpd -gq 2024-03-30T12:52:34.815 INFO:teuthology.orchestra.run.smithi067.stdout:30 Mar 12:52:34 ntpd[17196]: ---------------------------------------------------- 2024-03-30T12:52:34.815 INFO:teuthology.orchestra.run.smithi067.stdout:30 Mar 12:52:34 ntpd[17196]: ntp-4 is maintained by Network Time Foundation, 2024-03-30T12:52:34.815 INFO:teuthology.orchestra.run.smithi067.stdout:30 Mar 12:52:34 ntpd[17196]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-30T12:52:34.815 INFO:teuthology.orchestra.run.smithi067.stdout:30 Mar 12:52:34 ntpd[17196]: corporation. Support and training for ntp-4 are 2024-03-30T12:52:34.815 INFO:teuthology.orchestra.run.smithi067.stdout:30 Mar 12:52:34 ntpd[17196]: available at https://www.nwtime.org/support 2024-03-30T12:52:34.815 INFO:teuthology.orchestra.run.smithi067.stdout:30 Mar 12:52:34 ntpd[17196]: ---------------------------------------------------- 2024-03-30T12:52:34.816 INFO:teuthology.orchestra.run.smithi067.stdout:30 Mar 12:52:34 ntpd[17196]: proto: precision = 0.046 usec (-24) 2024-03-30T12:52:34.816 INFO:teuthology.orchestra.run.smithi067.stdout:30 Mar 12:52:34 ntpd[17196]: basedate set to 2022-02-04 2024-03-30T12:52:34.816 INFO:teuthology.orchestra.run.smithi067.stdout:30 Mar 12:52:34 ntpd[17196]: gps base set to 2022-02-06 (week 2196) 2024-03-30T12:52:34.816 INFO:teuthology.orchestra.run.smithi067.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-30T12:52:34.816 INFO:teuthology.orchestra.run.smithi067.stdout:30 Mar 12:52:34 ntpd[17196]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-30T12:52:34.816 INFO:teuthology.orchestra.run.smithi067.stdout:30 Mar 12:52:34 ntpd[17196]: restrict ::: KOD does nothing without LIMITED. 2024-03-30T12:52:34.817 INFO:teuthology.orchestra.run.smithi067.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-30T12:52:34.818 INFO:teuthology.orchestra.run.smithi067.stdout:30 Mar 12:52:34 ntpd[17196]: Listen and drop on 0 v6wildcard [::]:123 2024-03-30T12:52:34.818 INFO:teuthology.orchestra.run.smithi067.stdout:30 Mar 12:52:34 ntpd[17196]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-30T12:52:34.818 INFO:teuthology.orchestra.run.smithi067.stdout:30 Mar 12:52:34 ntpd[17196]: Listen normally on 2 lo 127.0.0.1:123 2024-03-30T12:52:34.818 INFO:teuthology.orchestra.run.smithi067.stdout:30 Mar 12:52:34 ntpd[17196]: Listen normally on 3 enp3s0f1 172.21.15.67:123 2024-03-30T12:52:34.818 INFO:teuthology.orchestra.run.smithi067.stdout:30 Mar 12:52:34 ntpd[17196]: Listen normally on 4 lo [::1]:123 2024-03-30T12:52:34.818 INFO:teuthology.orchestra.run.smithi067.stdout:30 Mar 12:52:34 ntpd[17196]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:feb6:fc0b%5]:123 2024-03-30T12:52:34.818 INFO:teuthology.orchestra.run.smithi067.stdout:30 Mar 12:52:34 ntpd[17196]: Listening on routing socket on fd #22 for interface updates 2024-03-30T12:52:34.837 INFO:teuthology.orchestra.run.smithi082.stdout:30 Mar 12:52:34 ntpd[17193]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-30T12:52:34.837 INFO:teuthology.orchestra.run.smithi082.stdout:30 Mar 12:52:34 ntpd[17193]: Command line: ntpd -gq 2024-03-30T12:52:34.837 INFO:teuthology.orchestra.run.smithi082.stdout:30 Mar 12:52:34 ntpd[17193]: ---------------------------------------------------- 2024-03-30T12:52:34.837 INFO:teuthology.orchestra.run.smithi082.stdout:30 Mar 12:52:34 ntpd[17193]: ntp-4 is maintained by Network Time Foundation, 2024-03-30T12:52:34.837 INFO:teuthology.orchestra.run.smithi082.stdout:30 Mar 12:52:34 ntpd[17193]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-30T12:52:34.837 INFO:teuthology.orchestra.run.smithi082.stdout:30 Mar 12:52:34 ntpd[17193]: corporation. Support and training for ntp-4 are 2024-03-30T12:52:34.838 INFO:teuthology.orchestra.run.smithi082.stdout:30 Mar 12:52:34 ntpd[17193]: available at https://www.nwtime.org/support 2024-03-30T12:52:34.838 INFO:teuthology.orchestra.run.smithi082.stdout:30 Mar 12:52:34 ntpd[17193]: ---------------------------------------------------- 2024-03-30T12:52:34.838 INFO:teuthology.orchestra.run.smithi082.stdout:30 Mar 12:52:34 ntpd[17193]: proto: precision = 0.062 usec (-24) 2024-03-30T12:52:34.838 INFO:teuthology.orchestra.run.smithi082.stdout:30 Mar 12:52:34 ntpd[17193]: basedate set to 2022-02-04 2024-03-30T12:52:34.838 INFO:teuthology.orchestra.run.smithi082.stdout:30 Mar 12:52:34 ntpd[17193]: gps base set to 2022-02-06 (week 2196) 2024-03-30T12:52:34.839 INFO:teuthology.orchestra.run.smithi082.stdout:30 Mar 12:52:34 ntpd[17193]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-30T12:52:34.839 INFO:teuthology.orchestra.run.smithi082.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-30T12:52:34.839 INFO:teuthology.orchestra.run.smithi082.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-30T12:52:34.840 INFO:teuthology.orchestra.run.smithi082.stdout:30 Mar 12:52:34 ntpd[17193]: restrict ::: KOD does nothing without LIMITED. 2024-03-30T12:52:34.840 INFO:teuthology.orchestra.run.smithi082.stdout:30 Mar 12:52:34 ntpd[17193]: Listen and drop on 0 v6wildcard [::]:123 2024-03-30T12:52:34.840 INFO:teuthology.orchestra.run.smithi082.stdout:30 Mar 12:52:34 ntpd[17193]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-30T12:52:34.840 INFO:teuthology.orchestra.run.smithi082.stdout:30 Mar 12:52:34 ntpd[17193]: Listen normally on 2 lo 127.0.0.1:123 2024-03-30T12:52:34.840 INFO:teuthology.orchestra.run.smithi082.stdout:30 Mar 12:52:34 ntpd[17193]: Listen normally on 3 enp3s0f1 172.21.15.82:123 2024-03-30T12:52:34.840 INFO:teuthology.orchestra.run.smithi082.stdout:30 Mar 12:52:34 ntpd[17193]: Listen normally on 4 lo [::1]:123 2024-03-30T12:52:34.840 INFO:teuthology.orchestra.run.smithi082.stdout:30 Mar 12:52:34 ntpd[17193]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d3ff%5]:123 2024-03-30T12:52:34.840 INFO:teuthology.orchestra.run.smithi082.stdout:30 Mar 12:52:34 ntpd[17193]: Listening on routing socket on fd #22 for interface updates 2024-03-30T12:52:35.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:35 ntpd[17193]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:35.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:35 ntpd[17193]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:35.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:35 ntpd[17193]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:35.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:35 ntpd[17193]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:35.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:35 ntpd[17193]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:35.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:35 ntpd[17193]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:35.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:35 ntpd[17193]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T12:52:35.839 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:35 ntpd[17193]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T12:52:36.815 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:36 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:36.815 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:36 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:37.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:37 ntpd[17193]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:37.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:37 ntpd[17193]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:37.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:37 ntpd[17193]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:37.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:37 ntpd[17193]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:37.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:37 ntpd[17193]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T12:52:37.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:37 ntpd[17193]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T12:52:38.815 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:38 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:38.815 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:38 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:38.815 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:38 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:38.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:38 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:38.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:38 ntpd[17196]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T12:52:38.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:38 ntpd[17196]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T12:52:38.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:38 ntpd[17193]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:38.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:38 ntpd[17193]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:38.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:38 ntpd[17193]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:38.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:38 ntpd[17193]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:38.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:38 ntpd[17193]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T12:52:38.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:38 ntpd[17193]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T12:52:39.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:39 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:39.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:39 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:39.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:39 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:39.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:39 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:39.817 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:39 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:39.817 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:39 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:39.817 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:39 ntpd[17196]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T12:52:39.817 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:39 ntpd[17196]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T12:52:39.817 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:39 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:39.817 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:39 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:39.817 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:39 ntpd[17196]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T12:52:39.817 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:39 ntpd[17196]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T12:52:39.839 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:39 ntpd[17193]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:39.839 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:39 ntpd[17193]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:39.839 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:39 ntpd[17193]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:39.839 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:39 ntpd[17193]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:39.840 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:39 ntpd[17193]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:39.840 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:39 ntpd[17193]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:39.840 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:39 ntpd[17193]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T12:52:39.840 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:39 ntpd[17193]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T12:52:39.840 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:39 ntpd[17193]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:39.840 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:39 ntpd[17193]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:39.840 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:39 ntpd[17193]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T12:52:39.840 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:39 ntpd[17193]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T12:52:40.815 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:40 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:40.815 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:40 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:40.815 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:40 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:40.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:40 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:40.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:40 ntpd[17196]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T12:52:40.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:40 ntpd[17196]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T12:52:40.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:40 ntpd[17193]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:40.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:40 ntpd[17193]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:40.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:40 ntpd[17193]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:40.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:40 ntpd[17193]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:40.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:40 ntpd[17193]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T12:52:40.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:40 ntpd[17193]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T12:52:41.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:41 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:41.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:41 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:41.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:41 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:41.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:41 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:41.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:41 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:41.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:41 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:41.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:41 ntpd[17196]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T12:52:41.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:41 ntpd[17196]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T12:52:41.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:41 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:41.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:41 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:41.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:41 ntpd[17196]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T12:52:41.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:41 ntpd[17196]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T12:52:41.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:41 ntpd[17193]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:41.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:41 ntpd[17193]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:41.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:41 ntpd[17193]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:41.838 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:41 ntpd[17193]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:41.839 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:41 ntpd[17193]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:41.839 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:41 ntpd[17193]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:41.839 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:41 ntpd[17193]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T12:52:41.839 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:41 ntpd[17193]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T12:52:41.839 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:41 ntpd[17193]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:41.839 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:41 ntpd[17193]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:41.839 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:41 ntpd[17193]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T12:52:41.839 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:41 ntpd[17193]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T12:52:41.839 INFO:teuthology.orchestra.run.smithi082.stdout:30 Mar 12:52:41 ntpd[17193]: ntpd: time slew -0.000821 s 2024-03-30T12:52:41.839 INFO:teuthology.orchestra.run.smithi082.stdout:ntpd: time slew -0.000821s 2024-03-30T12:52:41.840 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:41 ntpd[17193]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-30T12:52:41.840 INFO:teuthology.orchestra.run.smithi082.stderr:30 Mar 12:52:41 ntpd[17193]: can't open /var/log/ntpstats/loopstats.20240330: Permission denied 2024-03-30T12:52:41.890 INFO:teuthology.orchestra.run.smithi082.stdout: remote refid st t when poll reach delay offset jitter 2024-03-30T12:52:41.890 INFO:teuthology.orchestra.run.smithi082.stdout:============================================================================== 2024-03-30T12:52:41.890 INFO:teuthology.orchestra.run.smithi082.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-30T12:52:41.890 INFO:teuthology.orchestra.run.smithi082.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-30T12:52:41.890 INFO:teuthology.orchestra.run.smithi082.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-30T12:52:41.890 INFO:teuthology.orchestra.run.smithi082.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-30T12:52:42.815 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:42 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:42.815 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:42 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:42.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:42 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:42.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:42 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:42.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:42 ntpd[17196]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T12:52:42.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:42 ntpd[17196]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T12:52:43.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:43 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:43.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:43 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:43.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:43 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:43.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:43 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:43.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:43 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:43.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:43 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:43.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:43 ntpd[17196]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T12:52:43.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:43 ntpd[17196]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T12:52:43.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:43 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:43.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:43 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:43.817 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:43 ntpd[17196]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T12:52:43.817 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:43 ntpd[17196]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T12:52:44.815 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:44 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:44.815 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:44 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:44.815 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:44 ntpd[17196]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-30T12:52:44.815 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:44 ntpd[17196]: can't open /var/log/ntpstats/rawstats.20240330: Permission denied 2024-03-30T12:52:44.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:44 ntpd[17196]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-30T12:52:44.816 INFO:teuthology.orchestra.run.smithi067.stdout:30 Mar 12:52:44 ntpd[17196]: ntpd: time slew -0.001163 s 2024-03-30T12:52:44.816 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:44 ntpd[17196]: can't open /var/log/ntpstats/peerstats.20240330: Permission denied 2024-03-30T12:52:44.816 INFO:teuthology.orchestra.run.smithi067.stdout:ntpd: time slew -0.001163s 2024-03-30T12:52:44.817 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:44 ntpd[17196]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-30T12:52:44.817 INFO:teuthology.orchestra.run.smithi067.stderr:30 Mar 12:52:44 ntpd[17196]: can't open /var/log/ntpstats/loopstats.20240330: Permission denied 2024-03-30T12:52:44.867 INFO:teuthology.orchestra.run.smithi067.stdout: remote refid st t when poll reach delay offset jitter 2024-03-30T12:52:44.867 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================== 2024-03-30T12:52:44.867 INFO:teuthology.orchestra.run.smithi067.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-30T12:52:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-30T12:52:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-30T12:52:44.868 INFO:teuthology.orchestra.run.smithi067.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-30T12:52:44.869 INFO:teuthology.run_tasks:Running task install... 2024-03-30T12:52:44.878 DEBUG:teuthology.task.install:project ceph 2024-03-30T12:52:44.878 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc'}} 2024-03-30T12:52:44.878 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc'} 2024-03-30T12:52:44.879 INFO:teuthology.task.install:Using flavor: default 2024-03-30T12:52:44.884 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-30T12:52:44.884 INFO:teuthology.task.install:extra packages: [] 2024-03-30T12:52:44.885 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-key list | grep Ceph 2024-03-30T12:52:44.885 DEBUG:teuthology.orchestra.run.smithi082:> sudo apt-key list | grep Ceph 2024-03-30T12:52:44.947 INFO:teuthology.orchestra.run.smithi082.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-30T12:52:44.969 INFO:teuthology.orchestra.run.smithi067.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-30T12:52:44.975 INFO:teuthology.orchestra.run.smithi082.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-30T12:52:44.975 INFO:teuthology.orchestra.run.smithi082.stdout:uid [ unknown] Ceph.com (release key) 2024-03-30T12:52:44.975 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-03-30T12:52:44.976 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc 2024-03-30T12:52:45.000 INFO:teuthology.orchestra.run.smithi067.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-30T12:52:45.000 INFO:teuthology.orchestra.run.smithi067.stdout:uid [ unknown] Ceph.com (release key) 2024-03-30T12:52:45.000 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-03-30T12:52:45.000 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc 2024-03-30T12:52:45.121 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default/ 2024-03-30T12:52:45.122 INFO:teuthology.task.install.deb:Package version is 19.0.0-1864-g5dc2ea30-1jammy 2024-03-30T12:52:45.213 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default/ 2024-03-30T12:52:45.214 INFO:teuthology.task.install.deb:Package version is 19.0.0-1864-g5dc2ea30-1jammy 2024-03-30T12:52:45.399 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-30T12:52:45.399 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-30T12:52:45.413 DEBUG:teuthology.orchestra.run.smithi082:> sudo apt-get update 2024-03-30T12:52:45.490 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-30T12:52:45.490 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-30T12:52:45.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get update 2024-03-30T12:52:45.586 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-30T12:52:45.701 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-30T12:52:45.787 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-30T12:52:45.807 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-30T12:52:45.807 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-30T12:52:45.810 INFO:teuthology.orchestra.run.smithi082.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy InRelease 2024-03-30T12:52:45.865 INFO:teuthology.orchestra.run.smithi082.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-30T12:52:45.877 INFO:teuthology.orchestra.run.smithi082.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-30T12:52:45.878 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-30T12:52:45.916 INFO:teuthology.orchestra.run.smithi067.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy InRelease 2024-03-30T12:52:45.945 INFO:teuthology.orchestra.run.smithi082.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-30T12:52:45.960 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-30T12:52:45.983 INFO:teuthology.orchestra.run.smithi067.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-30T12:52:46.013 INFO:teuthology.orchestra.run.smithi082.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,412 B] 2024-03-30T12:52:46.052 INFO:teuthology.orchestra.run.smithi067.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-30T12:52:46.089 INFO:teuthology.orchestra.run.smithi082.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-03-30T12:52:46.121 INFO:teuthology.orchestra.run.smithi067.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,412 B] 2024-03-30T12:52:46.197 INFO:teuthology.orchestra.run.smithi067.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-03-30T12:52:46.247 INFO:teuthology.orchestra.run.smithi082.stdout:Fetched 28.9 kB in 1s (44.9 kB/s) 2024-03-30T12:52:46.285 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 28.9 kB in 1s (49.2 kB/s) 2024-03-30T12:52:47.237 INFO:teuthology.orchestra.run.smithi082.stdout:Reading package lists... 2024-03-30T12:52:47.253 DEBUG:teuthology.orchestra.run.smithi082:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-1864-g5dc2ea30-1jammy cephadm=19.0.0-1864-g5dc2ea30-1jammy ceph-mds=19.0.0-1864-g5dc2ea30-1jammy ceph-mgr=19.0.0-1864-g5dc2ea30-1jammy ceph-common=19.0.0-1864-g5dc2ea30-1jammy ceph-fuse=19.0.0-1864-g5dc2ea30-1jammy ceph-test=19.0.0-1864-g5dc2ea30-1jammy ceph-volume=19.0.0-1864-g5dc2ea30-1jammy radosgw=19.0.0-1864-g5dc2ea30-1jammy python3-rados=19.0.0-1864-g5dc2ea30-1jammy python3-rgw=19.0.0-1864-g5dc2ea30-1jammy python3-cephfs=19.0.0-1864-g5dc2ea30-1jammy python3-rbd=19.0.0-1864-g5dc2ea30-1jammy libcephfs2=19.0.0-1864-g5dc2ea30-1jammy libcephfs-dev=19.0.0-1864-g5dc2ea30-1jammy librados2=19.0.0-1864-g5dc2ea30-1jammy librbd1=19.0.0-1864-g5dc2ea30-1jammy rbd-fuse=19.0.0-1864-g5dc2ea30-1jammy 2024-03-30T12:52:47.265 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-03-30T12:52:47.281 DEBUG:teuthology.orchestra.run.smithi067:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-1864-g5dc2ea30-1jammy cephadm=19.0.0-1864-g5dc2ea30-1jammy ceph-mds=19.0.0-1864-g5dc2ea30-1jammy ceph-mgr=19.0.0-1864-g5dc2ea30-1jammy ceph-common=19.0.0-1864-g5dc2ea30-1jammy ceph-fuse=19.0.0-1864-g5dc2ea30-1jammy ceph-test=19.0.0-1864-g5dc2ea30-1jammy ceph-volume=19.0.0-1864-g5dc2ea30-1jammy radosgw=19.0.0-1864-g5dc2ea30-1jammy python3-rados=19.0.0-1864-g5dc2ea30-1jammy python3-rgw=19.0.0-1864-g5dc2ea30-1jammy python3-cephfs=19.0.0-1864-g5dc2ea30-1jammy python3-rbd=19.0.0-1864-g5dc2ea30-1jammy libcephfs2=19.0.0-1864-g5dc2ea30-1jammy libcephfs-dev=19.0.0-1864-g5dc2ea30-1jammy librados2=19.0.0-1864-g5dc2ea30-1jammy librbd1=19.0.0-1864-g5dc2ea30-1jammy rbd-fuse=19.0.0-1864-g5dc2ea30-1jammy 2024-03-30T12:52:47.316 INFO:teuthology.orchestra.run.smithi082.stdout:Reading package lists... 2024-03-30T12:52:47.345 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-03-30T12:52:47.514 INFO:teuthology.orchestra.run.smithi082.stdout:Building dependency tree... 2024-03-30T12:52:47.514 INFO:teuthology.orchestra.run.smithi082.stdout:Reading state information... 2024-03-30T12:52:47.547 INFO:teuthology.orchestra.run.smithi067.stdout:Building dependency tree... 2024-03-30T12:52:47.548 INFO:teuthology.orchestra.run.smithi067.stdout:Reading state information... 2024-03-30T12:52:47.706 INFO:teuthology.orchestra.run.smithi082.stdout:The following packages were automatically installed and are no longer required: 2024-03-30T12:52:47.706 INFO:teuthology.orchestra.run.smithi082.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-30T12:52:47.706 INFO:teuthology.orchestra.run.smithi082.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-30T12:52:47.706 INFO:teuthology.orchestra.run.smithi082.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-30T12:52:47.707 INFO:teuthology.orchestra.run.smithi082.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-30T12:52:47.707 INFO:teuthology.orchestra.run.smithi082.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-30T12:52:47.707 INFO:teuthology.orchestra.run.smithi082.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-30T12:52:47.707 INFO:teuthology.orchestra.run.smithi082.stdout:The following additional packages will be installed: 2024-03-30T12:52:47.707 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-30T12:52:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-30T12:52:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-30T12:52:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-30T12:52:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-30T12:52:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-30T12:52:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-30T12:52:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-30T12:52:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-30T12:52:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-30T12:52:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-30T12:52:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-30T12:52:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-30T12:52:47.709 INFO:teuthology.orchestra.run.smithi082.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-30T12:52:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-30T12:52:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-30T12:52:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-03-30T12:52:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-03-30T12:52:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-30T12:52:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-30T12:52:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-30T12:52:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-30T12:52:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-03-30T12:52:47.710 INFO:teuthology.orchestra.run.smithi082.stdout: xmlstarlet zip 2024-03-30T12:52:47.710 INFO:teuthology.orchestra.run.smithi082.stdout:Suggested packages: 2024-03-30T12:52:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-30T12:52:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-30T12:52:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-30T12:52:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-30T12:52:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-30T12:52:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-30T12:52:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-30T12:52:47.711 INFO:teuthology.orchestra.run.smithi082.stdout:Recommended packages: 2024-03-30T12:52:47.711 INFO:teuthology.orchestra.run.smithi082.stdout: btrfs-tools 2024-03-30T12:52:47.748 INFO:teuthology.orchestra.run.smithi067.stdout:The following packages were automatically installed and are no longer required: 2024-03-30T12:52:47.748 INFO:teuthology.orchestra.run.smithi067.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-30T12:52:47.748 INFO:teuthology.orchestra.run.smithi067.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-30T12:52:47.748 INFO:teuthology.orchestra.run.smithi067.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-30T12:52:47.749 INFO:teuthology.orchestra.run.smithi067.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-30T12:52:47.749 INFO:teuthology.orchestra.run.smithi067.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-30T12:52:47.749 INFO:teuthology.orchestra.run.smithi067.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-30T12:52:47.749 INFO:teuthology.orchestra.run.smithi067.stdout:The following additional packages will be installed: 2024-03-30T12:52:47.750 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-30T12:52:47.750 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-30T12:52:47.750 INFO:teuthology.orchestra.run.smithi067.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-30T12:52:47.750 INFO:teuthology.orchestra.run.smithi067.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-30T12:52:47.751 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-30T12:52:47.751 INFO:teuthology.orchestra.run.smithi067.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-30T12:52:47.751 INFO:teuthology.orchestra.run.smithi067.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-30T12:52:47.751 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-30T12:52:47.751 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-30T12:52:47.751 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-30T12:52:47.751 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-30T12:52:47.751 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-30T12:52:47.752 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-30T12:52:47.752 INFO:teuthology.orchestra.run.smithi067.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-30T12:52:47.752 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-30T12:52:47.752 INFO:teuthology.orchestra.run.smithi067.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-30T12:52:47.752 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-03-30T12:52:47.752 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-03-30T12:52:47.752 INFO:teuthology.orchestra.run.smithi067.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-30T12:52:47.752 INFO:teuthology.orchestra.run.smithi067.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-30T12:52:47.752 INFO:teuthology.orchestra.run.smithi067.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-30T12:52:47.752 INFO:teuthology.orchestra.run.smithi067.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-30T12:52:47.752 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-03-30T12:52:47.752 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet zip 2024-03-30T12:52:47.753 INFO:teuthology.orchestra.run.smithi067.stdout:Suggested packages: 2024-03-30T12:52:47.753 INFO:teuthology.orchestra.run.smithi067.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-30T12:52:47.753 INFO:teuthology.orchestra.run.smithi067.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-30T12:52:47.753 INFO:teuthology.orchestra.run.smithi067.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-30T12:52:47.753 INFO:teuthology.orchestra.run.smithi067.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-30T12:52:47.753 INFO:teuthology.orchestra.run.smithi067.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-30T12:52:47.753 INFO:teuthology.orchestra.run.smithi067.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-30T12:52:47.753 INFO:teuthology.orchestra.run.smithi067.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-30T12:52:47.753 INFO:teuthology.orchestra.run.smithi067.stdout:Recommended packages: 2024-03-30T12:52:47.753 INFO:teuthology.orchestra.run.smithi067.stdout: btrfs-tools 2024-03-30T12:52:47.812 INFO:teuthology.orchestra.run.smithi082.stdout:The following NEW packages will be installed: 2024-03-30T12:52:47.812 INFO:teuthology.orchestra.run.smithi082.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-30T12:52:47.812 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-30T12:52:47.812 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-03-30T12:52:47.812 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-03-30T12:52:47.812 INFO:teuthology.orchestra.run.smithi082.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-03-30T12:52:47.812 INFO:teuthology.orchestra.run.smithi082.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-03-30T12:52:47.813 INFO:teuthology.orchestra.run.smithi082.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-03-30T12:52:47.813 INFO:teuthology.orchestra.run.smithi082.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-03-30T12:52:47.814 INFO:teuthology.orchestra.run.smithi082.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-03-30T12:52:47.814 INFO:teuthology.orchestra.run.smithi082.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-03-30T12:52:47.814 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-03-30T12:52:47.814 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-30T12:52:47.814 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-30T12:52:47.814 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-30T12:52:47.814 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-30T12:52:47.814 INFO:teuthology.orchestra.run.smithi082.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-30T12:52:47.814 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-30T12:52:47.814 INFO:teuthology.orchestra.run.smithi082.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-30T12:52:47.814 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-03-30T12:52:47.814 INFO:teuthology.orchestra.run.smithi082.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-03-30T12:52:47.814 INFO:teuthology.orchestra.run.smithi082.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-03-30T12:52:47.814 INFO:teuthology.orchestra.run.smithi082.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-03-30T12:52:47.814 INFO:teuthology.orchestra.run.smithi082.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-03-30T12:52:47.814 INFO:teuthology.orchestra.run.smithi082.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-03-30T12:52:47.815 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-03-30T12:52:47.815 INFO:teuthology.orchestra.run.smithi082.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-03-30T12:52:47.865 INFO:teuthology.orchestra.run.smithi067.stdout:The following NEW packages will be installed: 2024-03-30T12:52:47.865 INFO:teuthology.orchestra.run.smithi067.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-30T12:52:47.865 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-30T12:52:47.865 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-03-30T12:52:47.865 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-03-30T12:52:47.865 INFO:teuthology.orchestra.run.smithi067.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-03-30T12:52:47.865 INFO:teuthology.orchestra.run.smithi067.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-03-30T12:52:47.866 INFO:teuthology.orchestra.run.smithi067.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-03-30T12:52:47.866 INFO:teuthology.orchestra.run.smithi067.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-03-30T12:52:47.866 INFO:teuthology.orchestra.run.smithi067.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-03-30T12:52:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-03-30T12:52:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-03-30T12:52:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-03-30T12:52:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-30T12:52:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-30T12:52:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-30T12:52:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-30T12:52:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-30T12:52:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-30T12:52:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-03-30T12:52:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-03-30T12:52:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-03-30T12:52:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-03-30T12:52:47.867 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-03-30T12:52:47.868 INFO:teuthology.orchestra.run.smithi067.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-03-30T12:52:47.868 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-03-30T12:52:47.868 INFO:teuthology.orchestra.run.smithi067.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-03-30T12:52:48.002 INFO:teuthology.orchestra.run.smithi082.stdout:0 upgraded, 118 newly installed, 0 to remove and 298 not upgraded. 2024-03-30T12:52:48.002 INFO:teuthology.orchestra.run.smithi082.stdout:Need to get 176 MB of archives. 2024-03-30T12:52:48.002 INFO:teuthology.orchestra.run.smithi082.stdout:After this operation, 730 MB of additional disk space will be used. 2024-03-30T12:52:48.002 INFO:teuthology.orchestra.run.smithi082.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-30T12:52:48.057 INFO:teuthology.orchestra.run.smithi067.stdout:0 upgraded, 118 newly installed, 0 to remove and 298 not upgraded. 2024-03-30T12:52:48.057 INFO:teuthology.orchestra.run.smithi067.stdout:Need to get 176 MB of archives. 2024-03-30T12:52:48.057 INFO:teuthology.orchestra.run.smithi067.stdout:After this operation, 730 MB of additional disk space will be used. 2024-03-30T12:52:48.057 INFO:teuthology.orchestra.run.smithi067.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-30T12:52:48.318 INFO:teuthology.orchestra.run.smithi082.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-1864-g5dc2ea30-1jammy [3,545 kB] 2024-03-30T12:52:48.376 INFO:teuthology.orchestra.run.smithi067.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-1864-g5dc2ea30-1jammy [3,545 kB] 2024-03-30T12:52:48.428 INFO:teuthology.orchestra.run.smithi082.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-30T12:52:48.441 INFO:teuthology.orchestra.run.smithi082.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-30T12:52:48.484 INFO:teuthology.orchestra.run.smithi067.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-30T12:52:48.495 INFO:teuthology.orchestra.run.smithi067.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-30T12:52:48.523 INFO:teuthology.orchestra.run.smithi082.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-30T12:52:48.578 INFO:teuthology.orchestra.run.smithi067.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-30T12:52:48.751 INFO:teuthology.orchestra.run.smithi082.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-30T12:52:48.764 INFO:teuthology.orchestra.run.smithi082.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-30T12:52:48.793 INFO:teuthology.orchestra.run.smithi082.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-30T12:52:48.802 INFO:teuthology.orchestra.run.smithi082.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-30T12:52:48.803 INFO:teuthology.orchestra.run.smithi082.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-30T12:52:48.805 INFO:teuthology.orchestra.run.smithi082.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-30T12:52:48.808 INFO:teuthology.orchestra.run.smithi067.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-30T12:52:48.820 INFO:teuthology.orchestra.run.smithi067.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-30T12:52:48.823 INFO:teuthology.orchestra.run.smithi082.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-30T12:52:48.849 INFO:teuthology.orchestra.run.smithi067.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-30T12:52:48.859 INFO:teuthology.orchestra.run.smithi067.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-30T12:52:48.860 INFO:teuthology.orchestra.run.smithi067.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-30T12:52:48.861 INFO:teuthology.orchestra.run.smithi067.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-30T12:52:48.880 INFO:teuthology.orchestra.run.smithi067.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-30T12:52:48.991 INFO:teuthology.orchestra.run.smithi082.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-30T12:52:49.048 INFO:teuthology.orchestra.run.smithi067.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-30T12:52:49.093 INFO:teuthology.orchestra.run.smithi067.stdout:Get:14 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-1864-g5dc2ea30-1jammy [3,220 kB] 2024-03-30T12:52:49.395 INFO:teuthology.orchestra.run.smithi082.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-30T12:52:49.429 INFO:teuthology.orchestra.run.smithi082.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-30T12:52:49.431 INFO:teuthology.orchestra.run.smithi082.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-30T12:52:49.451 INFO:teuthology.orchestra.run.smithi067.stdout:Get:15 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-1864-g5dc2ea30-1jammy [709 kB] 2024-03-30T12:52:49.461 INFO:teuthology.orchestra.run.smithi067.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-30T12:52:49.495 INFO:teuthology.orchestra.run.smithi067.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-30T12:52:49.497 INFO:teuthology.orchestra.run.smithi067.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-30T12:52:49.511 INFO:teuthology.orchestra.run.smithi082.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-30T12:52:49.524 INFO:teuthology.orchestra.run.smithi067.stdout:Get:19 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-1864-g5dc2ea30-1jammy [356 kB] 2024-03-30T12:52:49.537 INFO:teuthology.orchestra.run.smithi067.stdout:Get:20 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-1864-g5dc2ea30-1jammy [32.8 kB] 2024-03-30T12:52:49.539 INFO:teuthology.orchestra.run.smithi067.stdout:Get:21 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-1864-g5dc2ea30-1jammy [180 kB] 2024-03-30T12:52:49.562 INFO:teuthology.orchestra.run.smithi082.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-30T12:52:49.578 INFO:teuthology.orchestra.run.smithi067.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-30T12:52:49.584 INFO:teuthology.orchestra.run.smithi082.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-30T12:52:49.591 INFO:teuthology.orchestra.run.smithi082.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-30T12:52:49.592 INFO:teuthology.orchestra.run.smithi082.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-30T12:52:49.594 INFO:teuthology.orchestra.run.smithi067.stdout:Get:23 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-1864-g5dc2ea30-1jammy [64.6 kB] 2024-03-30T12:52:49.597 INFO:teuthology.orchestra.run.smithi067.stdout:Get:24 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-1864-g5dc2ea30-1jammy [330 kB] 2024-03-30T12:52:49.603 INFO:teuthology.orchestra.run.smithi082.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-30T12:52:49.604 INFO:teuthology.orchestra.run.smithi082.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-30T12:52:49.608 INFO:teuthology.orchestra.run.smithi067.stdout:Get:25 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-1864-g5dc2ea30-1jammy [6,575 kB] 2024-03-30T12:52:49.629 INFO:teuthology.orchestra.run.smithi067.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-30T12:52:49.651 INFO:teuthology.orchestra.run.smithi067.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-30T12:52:49.657 INFO:teuthology.orchestra.run.smithi067.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-30T12:52:49.659 INFO:teuthology.orchestra.run.smithi067.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-30T12:52:49.670 INFO:teuthology.orchestra.run.smithi067.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-30T12:52:49.671 INFO:teuthology.orchestra.run.smithi067.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-30T12:52:49.770 INFO:teuthology.orchestra.run.smithi082.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-30T12:52:49.836 INFO:teuthology.orchestra.run.smithi067.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-30T12:52:49.853 INFO:teuthology.orchestra.run.smithi082.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-30T12:52:49.881 INFO:teuthology.orchestra.run.smithi082.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-30T12:52:49.898 INFO:teuthology.orchestra.run.smithi082.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-30T12:52:49.919 INFO:teuthology.orchestra.run.smithi067.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-30T12:52:49.943 INFO:teuthology.orchestra.run.smithi082.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-30T12:52:49.948 INFO:teuthology.orchestra.run.smithi067.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-30T12:52:49.965 INFO:teuthology.orchestra.run.smithi067.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-30T12:52:50.009 INFO:teuthology.orchestra.run.smithi067.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-30T12:52:50.153 INFO:teuthology.orchestra.run.smithi082.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-30T12:52:50.168 INFO:teuthology.orchestra.run.smithi082.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-30T12:52:50.176 INFO:teuthology.orchestra.run.smithi082.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-30T12:52:50.181 INFO:teuthology.orchestra.run.smithi082.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-30T12:52:50.202 INFO:teuthology.orchestra.run.smithi082.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-30T12:52:50.205 INFO:teuthology.orchestra.run.smithi082.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-30T12:52:50.219 INFO:teuthology.orchestra.run.smithi067.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-30T12:52:50.235 INFO:teuthology.orchestra.run.smithi067.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-30T12:52:50.242 INFO:teuthology.orchestra.run.smithi067.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-30T12:52:50.246 INFO:teuthology.orchestra.run.smithi067.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-30T12:52:50.268 INFO:teuthology.orchestra.run.smithi067.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-30T12:52:50.271 INFO:teuthology.orchestra.run.smithi067.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-30T12:52:50.336 INFO:teuthology.orchestra.run.smithi067.stdout:Get:43 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-1864-g5dc2ea30-1jammy [112 kB] 2024-03-30T12:52:50.338 INFO:teuthology.orchestra.run.smithi067.stdout:Get:44 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-1864-g5dc2ea30-1jammy [466 kB] 2024-03-30T12:52:50.364 INFO:teuthology.orchestra.run.smithi067.stdout:Get:45 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-1864-g5dc2ea30-1jammy [24.2 MB] 2024-03-30T12:52:50.372 INFO:teuthology.orchestra.run.smithi082.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-30T12:52:50.438 INFO:teuthology.orchestra.run.smithi067.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-30T12:52:50.688 INFO:teuthology.orchestra.run.smithi082.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-30T12:52:50.727 INFO:teuthology.orchestra.run.smithi082.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-30T12:52:50.738 INFO:teuthology.orchestra.run.smithi082.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-30T12:52:50.754 INFO:teuthology.orchestra.run.smithi067.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-30T12:52:50.793 INFO:teuthology.orchestra.run.smithi067.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-30T12:52:50.804 INFO:teuthology.orchestra.run.smithi067.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-30T12:52:50.943 INFO:teuthology.orchestra.run.smithi082.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-30T12:52:50.943 INFO:teuthology.orchestra.run.smithi082.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-30T12:52:50.950 INFO:teuthology.orchestra.run.smithi082.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-30T12:52:50.955 INFO:teuthology.orchestra.run.smithi082.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-30T12:52:50.970 INFO:teuthology.orchestra.run.smithi082.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-30T12:52:50.991 INFO:teuthology.orchestra.run.smithi082.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-30T12:52:50.997 INFO:teuthology.orchestra.run.smithi082.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-30T12:52:51.008 INFO:teuthology.orchestra.run.smithi067.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-30T12:52:51.009 INFO:teuthology.orchestra.run.smithi067.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-30T12:52:51.016 INFO:teuthology.orchestra.run.smithi067.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-30T12:52:51.021 INFO:teuthology.orchestra.run.smithi067.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-30T12:52:51.036 INFO:teuthology.orchestra.run.smithi067.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-30T12:52:51.057 INFO:teuthology.orchestra.run.smithi067.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-30T12:52:51.063 INFO:teuthology.orchestra.run.smithi067.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-30T12:52:51.172 INFO:teuthology.orchestra.run.smithi082.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-30T12:52:51.238 INFO:teuthology.orchestra.run.smithi067.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-30T12:52:51.435 INFO:teuthology.orchestra.run.smithi082.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-30T12:52:51.491 INFO:teuthology.orchestra.run.smithi067.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-30T12:52:51.614 INFO:teuthology.orchestra.run.smithi082.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-30T12:52:51.627 INFO:teuthology.orchestra.run.smithi082.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-30T12:52:51.647 INFO:teuthology.orchestra.run.smithi082.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-30T12:52:51.668 INFO:teuthology.orchestra.run.smithi067.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-30T12:52:51.682 INFO:teuthology.orchestra.run.smithi067.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-30T12:52:51.702 INFO:teuthology.orchestra.run.smithi067.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-30T12:52:51.710 INFO:teuthology.orchestra.run.smithi082.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-30T12:52:51.770 INFO:teuthology.orchestra.run.smithi067.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-30T12:52:51.958 INFO:teuthology.orchestra.run.smithi082.stdout:Get:52 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-1864-g5dc2ea30-1jammy [3,220 kB] 2024-03-30T12:52:51.998 INFO:teuthology.orchestra.run.smithi082.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-30T12:52:52.000 INFO:teuthology.orchestra.run.smithi082.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-30T12:52:52.003 INFO:teuthology.orchestra.run.smithi082.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-30T12:52:52.003 INFO:teuthology.orchestra.run.smithi082.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-30T12:52:52.006 INFO:teuthology.orchestra.run.smithi082.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-30T12:52:52.057 INFO:teuthology.orchestra.run.smithi067.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-30T12:52:52.059 INFO:teuthology.orchestra.run.smithi067.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-30T12:52:52.062 INFO:teuthology.orchestra.run.smithi067.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-30T12:52:52.062 INFO:teuthology.orchestra.run.smithi067.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-30T12:52:52.065 INFO:teuthology.orchestra.run.smithi067.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-30T12:52:52.233 INFO:teuthology.orchestra.run.smithi082.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-30T12:52:52.287 INFO:teuthology.orchestra.run.smithi067.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-30T12:52:52.364 INFO:teuthology.orchestra.run.smithi067.stdout:Get:69 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-1864-g5dc2ea30-1jammy [5,081 kB] 2024-03-30T12:52:52.616 INFO:teuthology.orchestra.run.smithi067.stdout:Get:70 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-1864-g5dc2ea30-1jammy [243 kB] 2024-03-30T12:52:52.623 INFO:teuthology.orchestra.run.smithi067.stdout:Get:71 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-1864-g5dc2ea30-1jammy [124 kB] 2024-03-30T12:52:52.627 INFO:teuthology.orchestra.run.smithi067.stdout:Get:72 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-1864-g5dc2ea30-1jammy [1,521 kB] 2024-03-30T12:52:52.664 INFO:teuthology.orchestra.run.smithi082.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-30T12:52:52.669 INFO:teuthology.orchestra.run.smithi082.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-30T12:52:52.723 INFO:teuthology.orchestra.run.smithi067.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-30T12:52:52.727 INFO:teuthology.orchestra.run.smithi067.stdout:Get:74 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-1864-g5dc2ea30-1jammy [6,247 kB] 2024-03-30T12:52:52.729 INFO:teuthology.orchestra.run.smithi067.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-30T12:52:52.959 INFO:teuthology.orchestra.run.smithi082.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-30T12:52:52.959 INFO:teuthology.orchestra.run.smithi082.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-30T12:52:52.962 INFO:teuthology.orchestra.run.smithi082.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-30T12:52:52.963 INFO:teuthology.orchestra.run.smithi082.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-30T12:52:52.965 INFO:teuthology.orchestra.run.smithi082.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-30T12:52:52.967 INFO:teuthology.orchestra.run.smithi082.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-30T12:52:52.968 INFO:teuthology.orchestra.run.smithi082.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-30T12:52:52.969 INFO:teuthology.orchestra.run.smithi082.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-30T12:52:53.019 INFO:teuthology.orchestra.run.smithi067.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-30T12:52:53.019 INFO:teuthology.orchestra.run.smithi067.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-30T12:52:53.021 INFO:teuthology.orchestra.run.smithi067.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-30T12:52:53.023 INFO:teuthology.orchestra.run.smithi067.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-30T12:52:53.025 INFO:teuthology.orchestra.run.smithi067.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-30T12:52:53.027 INFO:teuthology.orchestra.run.smithi067.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-30T12:52:53.028 INFO:teuthology.orchestra.run.smithi067.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-30T12:52:53.030 INFO:teuthology.orchestra.run.smithi067.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-30T12:52:53.108 INFO:teuthology.orchestra.run.smithi067.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-1864-g5dc2ea30-1jammy [22.8 MB] 2024-03-30T12:52:53.151 INFO:teuthology.orchestra.run.smithi082.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-30T12:52:53.211 INFO:teuthology.orchestra.run.smithi067.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-30T12:52:53.559 INFO:teuthology.orchestra.run.smithi082.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-30T12:52:53.604 INFO:teuthology.orchestra.run.smithi082.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-30T12:52:53.623 INFO:teuthology.orchestra.run.smithi082.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-30T12:52:53.626 INFO:teuthology.orchestra.run.smithi067.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-30T12:52:53.654 INFO:teuthology.orchestra.run.smithi082.stdout:Get:73 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-1864-g5dc2ea30-1jammy [709 kB] 2024-03-30T12:52:53.672 INFO:teuthology.orchestra.run.smithi067.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-30T12:52:53.707 INFO:teuthology.orchestra.run.smithi067.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-30T12:52:53.736 INFO:teuthology.orchestra.run.smithi082.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-30T12:52:53.776 INFO:teuthology.orchestra.run.smithi082.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-30T12:52:53.788 INFO:teuthology.orchestra.run.smithi082.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-30T12:52:53.792 INFO:teuthology.orchestra.run.smithi082.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-30T12:52:53.805 INFO:teuthology.orchestra.run.smithi067.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-30T12:52:53.840 INFO:teuthology.orchestra.run.smithi082.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-30T12:52:53.843 INFO:teuthology.orchestra.run.smithi082.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-30T12:52:53.845 INFO:teuthology.orchestra.run.smithi067.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-30T12:52:53.856 INFO:teuthology.orchestra.run.smithi067.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-30T12:52:53.863 INFO:teuthology.orchestra.run.smithi067.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-30T12:52:53.864 INFO:teuthology.orchestra.run.smithi082.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-30T12:52:53.909 INFO:teuthology.orchestra.run.smithi067.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-30T12:52:53.912 INFO:teuthology.orchestra.run.smithi067.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-30T12:52:53.934 INFO:teuthology.orchestra.run.smithi067.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-30T12:52:54.028 INFO:teuthology.orchestra.run.smithi082.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-1864-g5dc2ea30-1jammy [356 kB] 2024-03-30T12:52:54.029 INFO:teuthology.orchestra.run.smithi082.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-30T12:52:54.098 INFO:teuthology.orchestra.run.smithi067.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-30T12:52:54.191 INFO:teuthology.orchestra.run.smithi082.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-1864-g5dc2ea30-1jammy [32.8 kB] 2024-03-30T12:52:54.192 INFO:teuthology.orchestra.run.smithi082.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-1864-g5dc2ea30-1jammy [180 kB] 2024-03-30T12:52:54.299 INFO:teuthology.orchestra.run.smithi082.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-1864-g5dc2ea30-1jammy [64.6 kB] 2024-03-30T12:52:54.301 INFO:teuthology.orchestra.run.smithi082.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-1864-g5dc2ea30-1jammy [330 kB] 2024-03-30T12:52:54.303 INFO:teuthology.orchestra.run.smithi082.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-30T12:52:54.321 INFO:teuthology.orchestra.run.smithi082.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-30T12:52:54.372 INFO:teuthology.orchestra.run.smithi067.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-30T12:52:54.390 INFO:teuthology.orchestra.run.smithi067.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-30T12:52:54.464 INFO:teuthology.orchestra.run.smithi082.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-1864-g5dc2ea30-1jammy [6,575 kB] 2024-03-30T12:52:54.474 INFO:teuthology.orchestra.run.smithi082.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-30T12:52:54.503 INFO:teuthology.orchestra.run.smithi082.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-30T12:52:54.544 INFO:teuthology.orchestra.run.smithi067.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-30T12:52:54.615 INFO:teuthology.orchestra.run.smithi067.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-30T12:52:54.617 INFO:teuthology.orchestra.run.smithi067.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-1864-g5dc2ea30-1jammy [14.0 kB] 2024-03-30T12:52:54.618 INFO:teuthology.orchestra.run.smithi067.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-1864-g5dc2ea30-1jammy [867 kB] 2024-03-30T12:52:54.636 INFO:teuthology.orchestra.run.smithi067.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-1864-g5dc2ea30-1jammy [2,190 kB] 2024-03-30T12:52:54.661 INFO:teuthology.orchestra.run.smithi082.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-30T12:52:54.664 INFO:teuthology.orchestra.run.smithi082.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-30T12:52:54.666 INFO:teuthology.orchestra.run.smithi082.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-30T12:52:54.688 INFO:teuthology.orchestra.run.smithi082.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-30T12:52:54.693 INFO:teuthology.orchestra.run.smithi082.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-30T12:52:54.722 INFO:teuthology.orchestra.run.smithi067.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-1864-g5dc2ea30-1jammy [777 kB] 2024-03-30T12:52:54.732 INFO:teuthology.orchestra.run.smithi067.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-30T12:52:54.734 INFO:teuthology.orchestra.run.smithi067.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-30T12:52:54.736 INFO:teuthology.orchestra.run.smithi067.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-30T12:52:54.758 INFO:teuthology.orchestra.run.smithi067.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-30T12:52:54.761 INFO:teuthology.orchestra.run.smithi067.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-1864-g5dc2ea30-1jammy [144 kB] 2024-03-30T12:52:54.763 INFO:teuthology.orchestra.run.smithi067.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-30T12:52:54.765 INFO:teuthology.orchestra.run.smithi067.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-1864-g5dc2ea30-1jammy [1,923 kB] 2024-03-30T12:52:54.871 INFO:teuthology.orchestra.run.smithi067.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-1864-g5dc2ea30-1jammy [8,625 kB] 2024-03-30T12:52:55.405 INFO:teuthology.orchestra.run.smithi067.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-1864-g5dc2ea30-1jammy [14.2 kB] 2024-03-30T12:52:55.405 INFO:teuthology.orchestra.run.smithi067.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-1864-g5dc2ea30-1jammy [49.9 MB] 2024-03-30T12:52:56.611 INFO:teuthology.orchestra.run.smithi082.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-1864-g5dc2ea30-1jammy [112 kB] 2024-03-30T12:52:56.623 INFO:teuthology.orchestra.run.smithi082.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-1864-g5dc2ea30-1jammy [466 kB] 2024-03-30T12:52:56.730 INFO:teuthology.orchestra.run.smithi082.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-1864-g5dc2ea30-1jammy [24.2 MB] 2024-03-30T12:52:57.802 INFO:teuthology.orchestra.run.smithi067.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-1864-g5dc2ea30-1jammy [121 kB] 2024-03-30T12:52:57.805 INFO:teuthology.orchestra.run.smithi067.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-1864-g5dc2ea30-1jammy [38.4 kB] 2024-03-30T12:52:57.806 INFO:teuthology.orchestra.run.smithi067.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-1864-g5dc2ea30-1jammy [13.0 MB] 2024-03-30T12:52:58.317 INFO:teuthology.orchestra.run.smithi067.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-1864-g5dc2ea30-1jammy [91.7 kB] 2024-03-30T12:52:58.913 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 176 MB in 10s (16.9 MB/s) 2024-03-30T12:52:59.082 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-30T12:52:59.186 INFO:teuthology.orchestra.run.smithi067.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-03-30T12:52:59.192 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-30T12:52:59.280 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-30T12:52:59.567 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-30T12:52:59.594 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-30T12:52:59.625 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-30T12:52:59.962 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-30T12:52:59.989 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-30T12:53:00.028 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-30T12:53:00.453 INFO:teuthology.orchestra.run.smithi082.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-1864-g5dc2ea30-1jammy [5,081 kB] 2024-03-30T12:53:00.473 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-30T12:53:00.500 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-30T12:53:00.549 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-30T12:53:00.820 INFO:teuthology.orchestra.run.smithi082.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-1864-g5dc2ea30-1jammy [243 kB] 2024-03-30T12:53:00.849 INFO:teuthology.orchestra.run.smithi082.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-1864-g5dc2ea30-1jammy [124 kB] 2024-03-30T12:53:00.852 INFO:teuthology.orchestra.run.smithi082.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-1864-g5dc2ea30-1jammy [1,521 kB] 2024-03-30T12:53:00.893 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-30T12:53:00.920 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-30T12:53:00.951 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-30T12:53:00.963 INFO:teuthology.orchestra.run.smithi082.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-1864-g5dc2ea30-1jammy [6,247 kB] 2024-03-30T12:53:01.279 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-30T12:53:01.306 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-30T12:53:01.346 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-30T12:53:01.419 INFO:teuthology.orchestra.run.smithi082.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-1864-g5dc2ea30-1jammy [22.8 MB] 2024-03-30T12:53:01.682 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-30T12:53:01.709 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-30T12:53:01.749 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-30T12:53:02.052 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package librados2. 2024-03-30T12:53:02.079 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../007-librados2_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:02.110 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking librados2 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:02.537 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package librbd1. 2024-03-30T12:53:02.564 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../008-librbd1_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:02.620 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking librbd1 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:02.780 INFO:teuthology.orchestra.run.smithi082.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-1864-g5dc2ea30-1jammy [14.0 kB] 2024-03-30T12:53:02.781 INFO:teuthology.orchestra.run.smithi082.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-1864-g5dc2ea30-1jammy [867 kB] 2024-03-30T12:53:02.798 INFO:teuthology.orchestra.run.smithi082.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-1864-g5dc2ea30-1jammy [2,190 kB] 2024-03-30T12:53:02.863 INFO:teuthology.orchestra.run.smithi082.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-1864-g5dc2ea30-1jammy [777 kB] 2024-03-30T12:53:02.897 INFO:teuthology.orchestra.run.smithi082.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-1864-g5dc2ea30-1jammy [144 kB] 2024-03-30T12:53:02.920 INFO:teuthology.orchestra.run.smithi082.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-1864-g5dc2ea30-1jammy [1,923 kB] 2024-03-30T12:53:02.999 INFO:teuthology.orchestra.run.smithi082.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-1864-g5dc2ea30-1jammy [8,625 kB] 2024-03-30T12:53:03.065 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libcephfs2. 2024-03-30T12:53:03.092 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:03.315 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libcephfs2 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:03.361 INFO:teuthology.orchestra.run.smithi082.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-1864-g5dc2ea30-1jammy [14.2 kB] 2024-03-30T12:53:03.362 INFO:teuthology.orchestra.run.smithi082.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-1864-g5dc2ea30-1jammy [49.9 MB] 2024-03-30T12:53:03.584 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-rados. 2024-03-30T12:53:03.600 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../010-python3-rados_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:03.651 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-rados (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:03.921 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-30T12:53:03.948 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T12:53:03.979 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-ceph-argparse (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:04.214 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-cephfs. 2024-03-30T12:53:04.231 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:04.272 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-cephfs (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:04.558 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-30T12:53:04.575 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T12:53:04.608 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-ceph-common (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:04.895 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-30T12:53:04.923 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-30T12:53:04.961 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-30T12:53:05.281 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-prettytable. 2024-03-30T12:53:05.308 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-03-30T12:53:05.338 INFO:teuthology.orchestra.run.smithi082.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-1864-g5dc2ea30-1jammy [121 kB] 2024-03-30T12:53:05.340 INFO:teuthology.orchestra.run.smithi082.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-1864-g5dc2ea30-1jammy [38.4 kB] 2024-03-30T12:53:05.341 INFO:teuthology.orchestra.run.smithi082.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-1864-g5dc2ea30-1jammy [13.0 MB] 2024-03-30T12:53:05.347 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-30T12:53:05.617 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-rbd. 2024-03-30T12:53:05.632 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:05.675 INFO:teuthology.orchestra.run.smithi082.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/wip-ceph_test_rados-partial-reads/5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-1864-g5dc2ea30-1jammy [91.7 kB] 2024-03-30T12:53:05.683 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-rbd (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:06.003 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-30T12:53:06.030 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-30T12:53:06.061 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-30T12:53:06.233 INFO:teuthology.orchestra.run.smithi082.stdout:Fetched 176 MB in 18s (9,862 kB/s) 2024-03-30T12:53:06.322 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-30T12:53:06.338 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-30T12:53:06.380 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-30T12:53:06.392 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-30T12:53:06.480 INFO:teuthology.orchestra.run.smithi082.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165997 files and directories currently installed.) 2024-03-30T12:53:06.485 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-30T12:53:06.566 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-30T12:53:06.700 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-30T12:53:06.716 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-30T12:53:06.758 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-30T12:53:06.911 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-30T12:53:06.937 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-30T12:53:06.978 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-30T12:53:07.128 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package lua5.1. 2024-03-30T12:53:07.155 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-30T12:53:07.194 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-30T12:53:07.297 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-30T12:53:07.324 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-30T12:53:07.365 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-30T12:53:07.464 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package lua-any. 2024-03-30T12:53:07.480 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-03-30T12:53:07.514 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-30T12:53:07.741 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package zip. 2024-03-30T12:53:07.757 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-03-30T12:53:07.767 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-30T12:53:07.793 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-30T12:53:07.799 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking zip (3.0-12build2) ... 2024-03-30T12:53:07.842 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-30T12:53:08.120 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package unzip. 2024-03-30T12:53:08.146 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-30T12:53:08.186 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-30T12:53:08.253 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-30T12:53:08.280 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-30T12:53:08.320 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-30T12:53:08.556 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package luarocks. 2024-03-30T12:53:08.583 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-30T12:53:08.614 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-30T12:53:08.632 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-30T12:53:08.658 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-30T12:53:08.698 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-30T12:53:08.997 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package librgw2. 2024-03-30T12:53:09.024 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../025-librgw2_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:09.051 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-30T12:53:09.063 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking librgw2 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:09.077 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-30T12:53:09.118 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-30T12:53:09.404 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package librados2. 2024-03-30T12:53:09.430 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../007-librados2_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:09.462 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking librados2 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:09.608 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-rgw. 2024-03-30T12:53:09.634 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:09.674 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-rgw (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:09.915 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package librbd1. 2024-03-30T12:53:09.941 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../008-librbd1_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:09.981 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking librbd1 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:10.018 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-30T12:53:10.045 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-30T12:53:10.085 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-30T12:53:10.388 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libradosstriper1. 2024-03-30T12:53:10.415 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:10.417 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libcephfs2. 2024-03-30T12:53:10.444 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:10.454 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libradosstriper1 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:10.484 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libcephfs2 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:10.761 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-rados. 2024-03-30T12:53:10.776 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../010-python3-rados_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:10.811 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-rados (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:10.883 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-common. 2024-03-30T12:53:10.910 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../029-ceph-common_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:10.949 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-common (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:11.072 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-30T12:53:11.088 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T12:53:11.264 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-ceph-argparse (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:11.508 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-cephfs. 2024-03-30T12:53:11.523 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:11.566 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-cephfs (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:11.861 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-30T12:53:11.887 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T12:53:11.927 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-ceph-common (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:12.255 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-30T12:53:12.279 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-base. 2024-03-30T12:53:12.281 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-30T12:53:12.306 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../030-ceph-base_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:12.322 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-30T12:53:12.355 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-base (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:12.600 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-prettytable. 2024-03-30T12:53:12.615 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-03-30T12:53:12.650 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-30T12:53:12.869 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-rbd. 2024-03-30T12:53:12.884 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:12.927 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-rbd (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:12.933 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-30T12:53:12.960 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-30T12:53:13.000 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-30T12:53:13.263 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-cheroot. 2024-03-30T12:53:13.288 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-30T12:53:13.290 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-30T12:53:13.314 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-30T12:53:13.355 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-30T12:53:13.458 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-30T12:53:13.666 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-30T12:53:13.681 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-30T12:53:13.724 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-30T12:53:13.849 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-30T12:53:13.876 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-30T12:53:13.916 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-30T12:53:14.010 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-30T12:53:14.025 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-30T12:53:14.060 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-30T12:53:14.244 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-30T12:53:14.270 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-30T12:53:14.310 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-30T12:53:14.380 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package lua5.1. 2024-03-30T12:53:14.406 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-30T12:53:14.446 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-30T12:53:14.563 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-30T12:53:14.590 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-30T12:53:14.621 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-30T12:53:14.741 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package lua-any. 2024-03-30T12:53:14.756 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-03-30T12:53:14.791 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-30T12:53:14.907 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-tempora. 2024-03-30T12:53:14.934 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-03-30T12:53:14.974 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-30T12:53:15.019 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package zip. 2024-03-30T12:53:15.045 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-03-30T12:53:15.085 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking zip (3.0-12build2) ... 2024-03-30T12:53:15.277 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-portend. 2024-03-30T12:53:15.304 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-03-30T12:53:15.343 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-30T12:53:15.413 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package unzip. 2024-03-30T12:53:15.439 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-30T12:53:15.480 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-30T12:53:15.629 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-30T12:53:15.656 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-30T12:53:15.696 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-30T12:53:15.834 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package luarocks. 2024-03-30T12:53:15.860 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-30T12:53:15.900 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-30T12:53:15.949 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-30T12:53:15.978 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-30T12:53:16.015 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-30T12:53:16.369 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-natsort. 2024-03-30T12:53:16.396 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-03-30T12:53:16.417 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package librgw2. 2024-03-30T12:53:16.436 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-30T12:53:16.443 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../025-librgw2_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:16.483 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking librgw2 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:16.722 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-logutils. 2024-03-30T12:53:16.738 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-03-30T12:53:16.772 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-30T12:53:17.060 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-rgw. 2024-03-30T12:53:17.075 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-30T12:53:17.087 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:17.102 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-30T12:53:17.127 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-rgw (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:17.141 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-30T12:53:17.444 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-mako. 2024-03-30T12:53:17.463 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-30T12:53:17.471 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-30T12:53:17.489 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-30T12:53:17.511 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-30T12:53:17.529 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-30T12:53:17.848 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-30T12:53:17.874 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-30T12:53:17.914 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-30T12:53:18.192 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-30T12:53:18.219 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-30T12:53:18.250 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-30T12:53:18.536 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-webob. 2024-03-30T12:53:18.563 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-30T12:53:18.575 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libradosstriper1. 2024-03-30T12:53:18.601 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:18.602 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-30T12:53:18.633 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libradosstriper1 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:18.956 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-waitress. 2024-03-30T12:53:18.983 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-30T12:53:19.025 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-30T12:53:19.070 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-common. 2024-03-30T12:53:19.096 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../029-ceph-common_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:19.136 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-common (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:19.333 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-tempita. 2024-03-30T12:53:19.360 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-30T12:53:19.400 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-30T12:53:19.686 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-paste. 2024-03-30T12:53:19.713 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-30T12:53:19.753 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-30T12:53:20.089 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-30T12:53:20.105 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-30T12:53:20.147 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-30T12:53:20.425 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-30T12:53:20.452 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-30T12:53:20.458 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-base. 2024-03-30T12:53:20.484 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../030-ceph-base_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:20.492 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-30T12:53:20.534 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-base (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:20.829 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-webtest. 2024-03-30T12:53:20.856 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-03-30T12:53:21.179 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-30T12:53:21.205 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-30T12:53:21.228 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-30T12:53:21.253 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-30T12:53:21.757 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-cheroot. 2024-03-30T12:53:21.783 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-30T12:53:21.823 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-30T12:53:21.865 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pecan. 2024-03-30T12:53:21.892 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-30T12:53:21.931 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-30T12:53:22.134 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-30T12:53:22.161 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-30T12:53:22.201 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-30T12:53:22.294 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-certifi. 2024-03-30T12:53:22.322 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-03-30T12:53:22.361 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-30T12:53:22.520 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-30T12:53:22.547 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-30T12:53:22.587 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-30T12:53:22.689 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-idna. 2024-03-30T12:53:22.716 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-03-30T12:53:22.756 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-30T12:53:22.957 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-30T12:53:22.983 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-30T12:53:23.032 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-30T12:53:23.033 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-urllib3. 2024-03-30T12:53:23.057 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-30T12:53:23.091 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-30T12:53:23.351 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-tempora. 2024-03-30T12:53:23.361 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-requests. 2024-03-30T12:53:23.377 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-30T12:53:23.378 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-03-30T12:53:23.417 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-30T12:53:23.419 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-30T12:53:23.704 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-portend. 2024-03-30T12:53:23.719 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-03-30T12:53:23.731 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-30T12:53:23.753 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-30T12:53:23.758 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-30T12:53:23.797 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-30T12:53:24.057 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-30T12:53:24.083 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-30T12:53:24.123 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-30T12:53:24.175 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-30T12:53:24.191 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T12:53:24.234 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-modules-core (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:24.451 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-30T12:53:24.477 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-30T12:53:24.517 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-30T12:53:24.613 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-30T12:53:24.641 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:24.680 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:24.854 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-natsort. 2024-03-30T12:53:24.881 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-03-30T12:53:24.921 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-30T12:53:24.941 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-30T12:53:24.968 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-30T12:53:24.999 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-30T12:53:25.232 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-logutils. 2024-03-30T12:53:25.259 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-03-30T12:53:25.299 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-30T12:53:25.302 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-openssl. 2024-03-30T12:53:25.329 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-03-30T12:53:25.369 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-30T12:53:25.585 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-30T12:53:25.611 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-30T12:53:25.651 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-30T12:53:25.755 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr. 2024-03-30T12:53:25.782 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:25.821 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:25.904 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-mako. 2024-03-30T12:53:25.931 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-30T12:53:25.974 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-30T12:53:26.150 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mon. 2024-03-30T12:53:26.166 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:26.200 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mon (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:26.349 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-30T12:53:26.376 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-30T12:53:26.424 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-30T12:53:26.795 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-osd. 2024-03-30T12:53:26.802 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-30T12:53:26.822 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:26.828 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-30T12:53:26.861 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-osd (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:26.868 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-30T12:53:27.171 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-webob. 2024-03-30T12:53:27.198 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-30T12:53:27.238 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-30T12:53:27.633 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-waitress. 2024-03-30T12:53:27.660 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-30T12:53:27.710 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-30T12:53:27.998 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph. 2024-03-30T12:53:28.025 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../066-ceph_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:28.064 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:28.110 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-tempita. 2024-03-30T12:53:28.137 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-30T12:53:28.185 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-30T12:53:28.342 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-fuse. 2024-03-30T12:53:28.369 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:28.400 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-fuse (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:28.521 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-paste. 2024-03-30T12:53:28.548 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-30T12:53:28.588 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-30T12:53:28.737 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mds. 2024-03-30T12:53:28.764 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:28.803 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mds (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:28.925 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-30T12:53:28.951 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-30T12:53:28.991 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-30T12:53:29.198 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package cephadm. 2024-03-30T12:53:29.225 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../069-cephadm_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:29.252 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-30T12:53:29.264 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking cephadm (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:29.267 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-30T12:53:29.302 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-30T12:53:29.572 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-webtest. 2024-03-30T12:53:29.593 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-30T12:53:29.598 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-03-30T12:53:29.619 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-30T12:53:29.638 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-30T12:53:29.659 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-30T12:53:29.978 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-30T12:53:29.994 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-30T12:53:30.000 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pecan. 2024-03-30T12:53:30.026 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-30T12:53:30.036 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-30T12:53:30.075 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-30T12:53:30.488 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-certifi. 2024-03-30T12:53:30.514 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-03-30T12:53:30.562 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-30T12:53:30.924 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-idna. 2024-03-30T12:53:30.950 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-03-30T12:53:30.990 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-30T12:53:31.058 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-babel. 2024-03-30T12:53:31.086 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-30T12:53:31.124 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-30T12:53:31.330 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-urllib3. 2024-03-30T12:53:31.357 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-30T12:53:31.401 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-30T12:53:31.444 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-jinja2. 2024-03-30T12:53:31.471 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-30T12:53:31.510 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-30T12:53:31.713 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-requests. 2024-03-30T12:53:31.739 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-30T12:53:31.779 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-30T12:53:31.856 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-30T12:53:31.883 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T12:53:31.922 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-cephadm (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:32.200 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-30T12:53:32.228 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-03-30T12:53:32.258 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-30T12:53:32.282 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-30T12:53:32.310 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-30T12:53:32.349 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-30T12:53:32.561 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-routes. 2024-03-30T12:53:32.588 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-30T12:53:32.652 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-30T12:53:32.652 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-30T12:53:32.654 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T12:53:32.693 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mgr-modules-core (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:32.964 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-30T12:53:32.992 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T12:53:33.031 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-dashboard (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:33.082 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-30T12:53:33.109 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:33.148 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:33.467 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-30T12:53:33.484 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-30T12:53:33.494 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-30T12:53:33.500 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-30T12:53:33.534 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-30T12:53:33.542 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-30T12:53:33.820 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-openssl. 2024-03-30T12:53:33.847 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-03-30T12:53:33.886 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-30T12:53:34.046 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-joblib. 2024-03-30T12:53:34.073 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-30T12:53:34.112 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-30T12:53:34.207 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mgr. 2024-03-30T12:53:34.234 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:34.264 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mgr (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:34.474 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-30T12:53:34.502 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-30T12:53:34.541 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-30T12:53:34.676 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mon. 2024-03-30T12:53:34.703 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:34.743 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mon (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:34.852 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-sklearn. 2024-03-30T12:53:34.879 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-30T12:53:34.918 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-30T12:53:35.396 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-osd. 2024-03-30T12:53:35.423 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:35.462 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-osd (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:35.653 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-30T12:53:35.680 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T12:53:35.812 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:36.491 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph. 2024-03-30T12:53:36.506 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../066-ceph_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:36.541 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:36.749 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-cachetools. 2024-03-30T12:53:36.776 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-03-30T12:53:36.815 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-30T12:53:36.818 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-fuse. 2024-03-30T12:53:36.845 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:36.885 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-fuse (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:37.160 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-30T12:53:37.188 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-30T12:53:37.226 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-30T12:53:37.413 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mds. 2024-03-30T12:53:37.440 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:37.479 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mds (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:37.546 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-30T12:53:37.574 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-30T12:53:37.613 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-30T12:53:37.875 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package cephadm. 2024-03-30T12:53:37.901 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../069-cephadm_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:37.933 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-rsa. 2024-03-30T12:53:37.941 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking cephadm (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:37.960 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-03-30T12:53:37.999 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-30T12:53:38.260 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-30T12:53:38.287 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-30T12:53:38.327 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-30T12:53:38.369 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-google-auth. 2024-03-30T12:53:38.397 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-03-30T12:53:38.446 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-30T12:53:38.663 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-30T12:53:38.689 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-30T12:53:38.721 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-30T12:53:38.831 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-30T12:53:38.863 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-30T12:53:38.906 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-30T12:53:39.284 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-websocket. 2024-03-30T12:53:39.311 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-03-30T12:53:39.359 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-30T12:53:39.843 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-babel. 2024-03-30T12:53:39.870 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-30T12:53:39.909 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-30T12:53:39.920 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-30T12:53:39.948 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-30T12:53:39.987 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-30T12:53:40.229 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-jinja2. 2024-03-30T12:53:40.256 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-30T12:53:40.298 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-30T12:53:40.665 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-30T12:53:40.692 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T12:53:40.692 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-30T12:53:40.709 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T12:53:40.731 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mgr-cephadm (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:40.750 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:41.068 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-30T12:53:41.095 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-30T12:53:41.095 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-03-30T12:53:41.123 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-30T12:53:41.134 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-30T12:53:41.161 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-30T12:53:41.437 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-routes. 2024-03-30T12:53:41.447 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-30T12:53:41.464 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-30T12:53:41.464 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-30T12:53:41.497 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-30T12:53:41.503 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-30T12:53:41.767 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package jq. 2024-03-30T12:53:41.784 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-30T12:53:41.825 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-30T12:53:41.857 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-30T12:53:41.884 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T12:53:41.923 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mgr-dashboard (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:42.078 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package socat. 2024-03-30T12:53:42.106 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-30T12:53:42.144 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-30T12:53:42.448 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package xmlstarlet. 2024-03-30T12:53:42.464 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-30T12:53:42.506 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-30T12:53:42.535 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-30T12:53:42.562 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-30T12:53:42.601 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-30T12:53:42.868 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-test. 2024-03-30T12:53:42.885 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../097-ceph-test_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:42.926 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-test (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:43.130 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-joblib. 2024-03-30T12:53:43.157 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-30T12:53:43.196 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-30T12:53:43.508 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-30T12:53:43.535 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-30T12:53:43.566 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-30T12:53:43.861 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-sklearn. 2024-03-30T12:53:43.888 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-30T12:53:43.927 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-30T12:53:44.646 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-30T12:53:44.673 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T12:53:44.798 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:45.189 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-volume. 2024-03-30T12:53:45.217 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T12:53:45.256 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-volume (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:45.526 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-30T12:53:45.543 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:45.576 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libcephfs-dev (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:45.768 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-cachetools. 2024-03-30T12:53:45.795 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-03-30T12:53:45.834 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-30T12:53:45.887 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-30T12:53:45.915 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-30T12:53:45.954 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-30T12:53:46.162 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-30T12:53:46.189 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-30T12:53:46.228 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-30T12:53:46.300 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-30T12:53:46.328 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-03-30T12:53:46.366 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-30T12:53:46.532 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-30T12:53:46.547 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-30T12:53:46.581 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-30T12:53:46.695 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package nvme-cli. 2024-03-30T12:53:46.723 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-30T12:53:46.762 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-30T12:53:46.868 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-rsa. 2024-03-30T12:53:46.895 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-03-30T12:53:46.926 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-30T12:53:47.184 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package pkg-config. 2024-03-30T12:53:47.212 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-30T12:53:47.230 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-google-auth. 2024-03-30T12:53:47.251 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-30T12:53:47.257 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-03-30T12:53:47.296 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-30T12:53:47.511 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-30T12:53:47.539 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-30T12:53:47.578 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-30T12:53:47.615 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-30T12:53:47.643 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-30T12:53:47.691 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-30T12:53:47.952 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-websocket. 2024-03-30T12:53:47.979 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-03-30T12:53:48.014 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-30T12:53:48.018 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-30T12:53:48.042 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-30T12:53:48.081 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-30T12:53:48.296 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-30T12:53:48.324 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-30T12:53:48.354 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-30T12:53:48.392 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pastescript. 2024-03-30T12:53:48.420 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-03-30T12:53:48.458 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-30T12:53:48.803 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pluggy. 2024-03-30T12:53:48.831 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-30T12:53:48.870 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-30T12:53:49.068 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-30T12:53:49.085 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T12:53:49.126 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:49.131 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-psutil. 2024-03-30T12:53:49.159 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-30T12:53:49.189 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-30T12:53:49.412 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-30T12:53:49.440 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-30T12:53:49.470 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-30T12:53:49.509 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-py. 2024-03-30T12:53:49.537 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-03-30T12:53:49.575 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-30T12:53:49.832 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-30T12:53:49.853 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pygments. 2024-03-30T12:53:49.859 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-30T12:53:49.870 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-30T12:53:49.898 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-30T12:53:49.903 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-30T12:53:50.168 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package jq. 2024-03-30T12:53:50.183 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-30T12:53:50.226 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-30T12:53:50.332 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-30T12:53:50.361 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-30T12:53:50.399 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-30T12:53:50.470 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package socat. 2024-03-30T12:53:50.486 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-30T12:53:50.520 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-30T12:53:50.660 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-toml. 2024-03-30T12:53:50.688 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-03-30T12:53:50.726 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-30T12:53:50.798 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package xmlstarlet. 2024-03-30T12:53:50.814 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-30T12:53:50.857 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-30T12:53:51.029 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pytest. 2024-03-30T12:53:51.057 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-30T12:53:51.095 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-30T12:53:51.219 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-test. 2024-03-30T12:53:51.234 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../097-ceph-test_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:51.277 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-test (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:51.407 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-simplejson. 2024-03-30T12:53:51.436 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-30T12:53:51.465 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-30T12:53:51.710 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-30T12:53:51.727 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-30T12:53:51.768 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-30T12:53:52.522 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package radosgw. 2024-03-30T12:53:52.550 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../116-radosgw_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:52.588 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking radosgw (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:53.444 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package rbd-fuse. 2024-03-30T12:53:53.472 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:53.502 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking rbd-fuse (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:53.622 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package ceph-volume. 2024-03-30T12:53:53.650 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-1864-g5dc2ea30-1jammy_all.deb ... 2024-03-30T12:53:53.697 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking ceph-volume (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:53.798 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-30T12:53:54.026 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-30T12:53:54.053 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:53:54.056 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-30T12:53:54.101 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking libcephfs-dev (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:54.183 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-30T12:53:54.450 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-30T12:53:54.453 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-30T12:53:54.481 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-30T12:53:54.520 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-30T12:53:54.800 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-30T12:53:54.875 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-30T12:53:54.902 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-03-30T12:53:54.941 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-30T12:53:55.249 INFO:teuthology.orchestra.run.smithi067.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-30T12:53:55.318 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up cephadm (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:53:55.353 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package nvme-cli. 2024-03-30T12:53:55.380 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-30T12:53:55.428 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-30T12:53:55.866 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package pkg-config. 2024-03-30T12:53:55.892 INFO:teuthology.orchestra.run.smithi067.stdout:Adding system user cephadm....done 2024-03-30T12:53:55.893 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-30T12:53:55.932 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-30T12:53:56.049 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-30T12:53:56.160 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-30T12:53:56.176 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-30T12:53:56.210 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-30T12:53:56.347 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-30T12:53:56.626 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-30T12:53:56.638 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-30T12:53:56.666 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-30T12:53:56.713 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-30T12:53:56.752 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-30T12:53:56.990 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pastescript. 2024-03-30T12:53:57.018 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-03-30T12:53:57.065 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-30T12:53:57.070 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-30T12:53:57.329 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-30T12:53:57.385 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pluggy. 2024-03-30T12:53:57.413 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-30T12:53:57.439 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-30T12:53:57.452 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-30T12:53:57.746 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-psutil. 2024-03-30T12:53:57.765 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-30T12:53:57.773 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-30T12:53:57.804 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-30T12:53:58.099 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-py. 2024-03-30T12:53:58.126 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-03-30T12:53:58.142 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-30T12:53:58.165 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-30T12:53:58.410 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-30T12:53:58.443 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pygments. 2024-03-30T12:53:58.459 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-30T12:53:58.493 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-30T12:53:58.678 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-30T12:53:58.921 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-30T12:53:58.972 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-30T12:53:59.000 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-30T12:53:59.038 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-30T12:53:59.189 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-30T12:53:59.308 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-toml. 2024-03-30T12:53:59.324 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-03-30T12:53:59.358 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-30T12:53:59.465 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-30T12:53:59.627 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-pytest. 2024-03-30T12:53:59.655 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-30T12:53:59.693 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-30T12:53:59.783 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-ceph-argparse (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:00.047 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package python3-simplejson. 2024-03-30T12:54:00.059 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-30T12:54:00.062 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-30T12:54:00.105 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-30T12:54:00.210 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-30T12:54:00.328 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-30T12:54:00.416 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-30T12:54:00.429 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-30T12:54:00.432 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-30T12:54:00.474 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-30T12:54:00.680 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-30T12:54:01.101 INFO:teuthology.orchestra.run.smithi067.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-30T12:54:01.137 INFO:teuthology.orchestra.run.smithi067.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-30T12:54:01.225 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-30T12:54:01.236 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package radosgw. 2024-03-30T12:54:01.264 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../116-radosgw_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:54:01.302 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking radosgw (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:01.359 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-30T12:54:01.727 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-30T12:54:01.836 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-30T12:54:02.064 INFO:teuthology.orchestra.run.smithi082.stdout:Selecting previously unselected package rbd-fuse. 2024-03-30T12:54:02.080 INFO:teuthology.orchestra.run.smithi082.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-1864-g5dc2ea30-1jammy_amd64.deb ... 2024-03-30T12:54:02.087 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-30T12:54:02.122 INFO:teuthology.orchestra.run.smithi082.stdout:Unpacking rbd-fuse (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:02.347 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-30T12:54:02.434 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-30T12:54:02.673 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up zip (3.0-12build2) ... 2024-03-30T12:54:02.685 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-30T12:54:02.799 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-30T12:54:02.811 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-30T12:54:03.074 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-30T12:54:03.440 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-30T12:54:03.484 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-30T12:54:03.752 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-30T12:54:03.867 INFO:teuthology.orchestra.run.smithi082.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-30T12:54:03.878 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-30T12:54:03.938 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up cephadm (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:04.024 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-30T12:54:04.364 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-30T12:54:04.766 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-30T12:54:04.779 INFO:teuthology.orchestra.run.smithi082.stdout:Adding system user cephadm....done 2024-03-30T12:54:04.919 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-30T12:54:05.058 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-30T12:54:05.204 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-30T12:54:05.443 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-30T12:54:05.446 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-30T12:54:05.556 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-30T12:54:05.736 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-30T12:54:05.824 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-30T12:54:06.080 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-30T12:54:06.092 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-30T12:54:06.218 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-30T12:54:06.339 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-30T12:54:06.456 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-ceph-common (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:06.519 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-30T12:54:06.774 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-idna (3.3-1) ... 2024-03-30T12:54:06.881 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-30T12:54:07.092 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-30T12:54:07.114 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-30T12:54:07.398 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-30T12:54:07.658 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-30T12:54:07.910 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-30T12:54:08.168 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-30T12:54:08.314 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-30T12:54:08.558 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-30T12:54:08.586 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-ceph-argparse (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:08.700 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-30T12:54:08.845 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-30T12:54:08.968 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-30T12:54:08.971 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-30T12:54:09.094 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-30T12:54:09.097 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-30T12:54:09.220 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-30T12:54:09.232 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-30T12:54:09.513 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-30T12:54:09.524 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-30T12:54:09.789 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-30T12:54:09.904 INFO:teuthology.orchestra.run.smithi082.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-30T12:54:09.938 INFO:teuthology.orchestra.run.smithi082.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-30T12:54:09.994 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-30T12:54:10.066 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-30T12:54:10.095 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-30T12:54:10.217 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-30T12:54:10.463 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-30T12:54:10.518 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-30T12:54:10.589 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-30T12:54:10.786 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-30T12:54:10.857 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-30T12:54:11.037 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-30T12:54:11.124 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-30T12:54:11.288 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-30T12:54:11.442 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up zip (3.0-12build2) ... 2024-03-30T12:54:11.539 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-30T12:54:11.568 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-30T12:54:11.815 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-30T12:54:11.941 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-30T12:54:12.203 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-30T12:54:12.234 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-30T12:54:12.360 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-30T12:54:12.454 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-30T12:54:12.580 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-30T12:54:12.662 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-30T12:54:12.706 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-30T12:54:12.955 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-30T12:54:12.990 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-30T12:54:13.214 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-30T12:54:13.331 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-30T12:54:13.374 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-30T12:54:13.458 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-30T12:54:13.546 INFO:teuthology.orchestra.run.smithi067.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-30T12:54:13.626 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-30T12:54:13.784 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-30T12:54:13.919 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-30T12:54:13.985 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-30T12:54:14.270 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-30T12:54:14.320 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-30T12:54:14.580 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-30T12:54:14.613 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-30T12:54:14.731 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-30T12:54:14.864 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-30T12:54:14.998 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-ceph-common (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:15.015 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-30T12:54:15.141 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-30T12:54:15.291 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-idna (3.3-1) ... 2024-03-30T12:54:15.484 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-30T12:54:15.609 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-30T12:54:15.852 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-30T12:54:16.137 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-30T12:54:16.238 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-30T12:54:16.522 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-30T12:54:16.798 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-30T12:54:16.806 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-30T12:54:17.057 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-30T12:54:17.160 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-30T12:54:17.192 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-30T12:54:17.468 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-30T12:54:17.553 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-30T12:54:17.594 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-30T12:54:17.720 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-30T12:54:17.863 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-30T12:54:18.021 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-30T12:54:18.207 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-30T12:54:18.280 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-30T12:54:18.358 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up librados2 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:18.484 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libsqlite3-mod-ceph (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:18.540 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-30T12:54:18.610 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-30T12:54:18.666 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-30T12:54:18.946 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libcephfs2 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:18.975 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-30T12:54:19.089 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libradosstriper1 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:19.218 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-30T12:54:19.240 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-30T12:54:19.469 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-30T12:54:19.550 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up librbd1 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:19.676 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-modules-core (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:19.711 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-30T12:54:19.802 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-fuse (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:19.946 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-30T12:54:20.034 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-30T12:54:20.035 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-30T12:54:20.206 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-30T12:54:20.340 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-30T12:54:20.512 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libcephfs-dev (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:20.618 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-30T12:54:20.638 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-rados (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:20.742 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-30T12:54:20.772 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up librgw2 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:20.923 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-30T12:54:21.035 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-30T12:54:21.352 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-30T12:54:21.595 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-30T12:54:21.721 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-30T12:54:21.880 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-30T12:54:21.970 INFO:teuthology.orchestra.run.smithi082.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-30T12:54:22.193 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-rbd (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:22.223 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-30T12:54:22.320 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up rbd-fuse (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:22.349 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-30T12:54:22.446 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-rgw (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:22.572 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-cephfs (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:22.723 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-common (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:22.751 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-30T12:54:22.985 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-30T12:54:23.086 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-30T12:54:23.353 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-30T12:54:23.432 INFO:teuthology.orchestra.run.smithi067.stdout:Adding group ceph....done 2024-03-30T12:54:23.479 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-30T12:54:23.814 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-30T12:54:23.818 INFO:teuthology.orchestra.run.smithi067.stdout:Adding system user ceph....done 2024-03-30T12:54:24.011 INFO:teuthology.orchestra.run.smithi067.stdout:Setting system user ceph properties....done 2024-03-30T12:54:24.028 INFO:teuthology.orchestra.run.smithi067.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-30T12:54:24.123 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-30T12:54:24.176 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-30T12:54:24.399 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-30T12:54:24.483 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-30T12:54:24.525 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-30T12:54:24.801 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-30T12:54:24.958 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-test (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:25.083 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up radosgw (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:25.102 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-30T12:54:25.464 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-30T12:54:25.559 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-30T12:54:25.559 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-30T12:54:25.841 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-30T12:54:26.133 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-30T12:54:26.452 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-30T12:54:26.498 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-base (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:26.579 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up librados2 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:26.704 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libsqlite3-mod-ceph (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:26.839 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-30T12:54:26.899 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-30T12:54:27.175 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libcephfs2 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:27.276 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libradosstriper1 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:27.377 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-30T12:54:27.391 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mds (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:27.634 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-30T12:54:27.635 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-30T12:54:27.645 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up librbd1 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:27.771 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mgr-modules-core (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:27.872 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-fuse (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:28.072 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-30T12:54:28.072 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-30T12:54:28.101 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:28.335 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-30T12:54:28.335 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-30T12:54:28.812 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-osd (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:28.849 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up libcephfs-dev (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:29.008 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-rados (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:29.102 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-30T12:54:29.103 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-30T12:54:29.134 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up librgw2 (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:29.260 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-30T12:54:29.571 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-k8sevents (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:29.706 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:29.882 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mon (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:30.130 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-30T12:54:30.130 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-30T12:54:30.471 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-rbd (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:30.589 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up rbd-fuse (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:30.600 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-cephadm (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:30.698 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-rgw (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:30.824 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up python3-cephfs (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:30.934 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:30.950 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-common (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:31.035 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-dashboard (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:31.170 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-volume (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:31.493 INFO:teuthology.orchestra.run.smithi082.stdout:Adding group ceph....done 2024-03-30T12:54:31.514 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-30T12:54:31.623 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-30T12:54:31.775 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-30T12:54:31.878 INFO:teuthology.orchestra.run.smithi082.stdout:Adding system user ceph....done 2024-03-30T12:54:32.021 INFO:teuthology.orchestra.run.smithi082.stdout:Setting system user ceph properties....done 2024-03-30T12:54:32.040 INFO:teuthology.orchestra.run.smithi082.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-30T12:54:32.178 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-30T12:54:32.483 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-30T12:54:32.801 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-30T12:54:32.951 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-test (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:33.077 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up radosgw (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:33.476 INFO:teuthology.orchestra.run.smithi067.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-30T12:54:33.482 DEBUG:teuthology.parallel:result is None 2024-03-30T12:54:33.514 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-30T12:54:33.515 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-30T12:54:33.978 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-base (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:34.312 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-30T12:54:34.766 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mds (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:34.989 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-30T12:54:34.989 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-30T12:54:35.459 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mgr (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:35.688 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-30T12:54:35.688 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-30T12:54:36.144 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-osd (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:36.415 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-30T12:54:36.415 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-30T12:54:36.895 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mgr-k8sevents (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:37.004 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:37.131 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mon (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:37.330 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-30T12:54:37.330 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-30T12:54:37.790 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mgr-cephadm (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:37.925 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:38.051 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-mgr-dashboard (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:38.210 INFO:teuthology.orchestra.run.smithi082.stdout:Setting up ceph-volume (19.0.0-1864-g5dc2ea30-1jammy) ... 2024-03-30T12:54:38.554 INFO:teuthology.orchestra.run.smithi082.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-30T12:54:38.680 INFO:teuthology.orchestra.run.smithi082.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-30T12:54:38.833 INFO:teuthology.orchestra.run.smithi082.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-30T12:54:39.917 INFO:teuthology.orchestra.run.smithi082.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-30T12:54:40.513 INFO:teuthology.orchestra.run.smithi082.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-30T12:54:40.518 DEBUG:teuthology.parallel:result is None 2024-03-30T12:54:40.518 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc 2024-03-30T12:54:40.738 DEBUG:teuthology.orchestra.run.smithi067:> dpkg-query -W -f '${Version}' ceph 2024-03-30T12:54:40.758 INFO:teuthology.orchestra.run.smithi067.stdout:19.0.0-1864-g5dc2ea30-1jammy 2024-03-30T12:54:40.758 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1864-g5dc2ea30-1jammy 2024-03-30T12:54:40.759 INFO:teuthology.task.install:The correct ceph version 19.0.0-1864-g5dc2ea30-1jammy is installed. 2024-03-30T12:54:40.760 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc 2024-03-30T12:54:40.902 DEBUG:teuthology.orchestra.run.smithi082:> dpkg-query -W -f '${Version}' ceph 2024-03-30T12:54:40.921 INFO:teuthology.orchestra.run.smithi082.stdout:19.0.0-1864-g5dc2ea30-1jammy 2024-03-30T12:54:40.921 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1864-g5dc2ea30-1jammy 2024-03-30T12:54:40.921 INFO:teuthology.task.install:The correct ceph version 19.0.0-1864-g5dc2ea30-1jammy is installed. 2024-03-30T12:54:40.923 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-30T12:54:40.923 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-30T12:54:40.923 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-30T12:54:40.938 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-30T12:54:40.938 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-30T12:54:40.977 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-30T12:54:40.977 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-30T12:54:40.977 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/daemon-helper 2024-03-30T12:54:40.992 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-30T12:54:41.048 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-30T12:54:41.048 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/daemon-helper 2024-03-30T12:54:41.061 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-30T12:54:41.116 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-30T12:54:41.116 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-30T12:54:41.116 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-30T12:54:41.131 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-30T12:54:41.183 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-30T12:54:41.184 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-30T12:54:41.196 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-30T12:54:41.252 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-30T12:54:41.252 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-30T12:54:41.252 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/stdin-killer 2024-03-30T12:54:41.265 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-30T12:54:41.319 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-30T12:54:41.320 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/stdin-killer 2024-03-30T12:54:41.332 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-30T12:54:41.388 INFO:teuthology.run_tasks:Running task ceph... 2024-03-30T12:54:41.485 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-30T12:54:41.486 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 777 /var/log/ceph 2024-03-30T12:54:41.488 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 777 /var/log/ceph 2024-03-30T12:54:41.501 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-30T12:54:41.501 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-30T12:54:41.546 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-30T12:54:41.558 INFO:tasks.ceph:Creating extra log directories... 2024-03-30T12:54:41.559 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-30T12:54:41.602 DEBUG:teuthology.orchestra.run.smithi082:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-30T12:54:41.625 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-30T12:54:41.626 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 2, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': '*', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-30T12:54:41.626 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630730', 'branch': 'wip-ceph_test_rados-partial-reads', 'description': 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-snaps-few-objects-overwrites}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7630730', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-ceph_test_rados-partial-reads'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': '*', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 2, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'create_rbd_pool': False, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'sha1': '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc'}}, 'rados': {'wait_for_all_active_clean_pgs': True}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5, 'disable_objectstore_tool_tests': True}, 'workunit': {'branch': 'wip-ceph_test_rados-partial-reads', 'sha1': '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 9805, 'sha1': '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-ceph_test_rados-partial-reads', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc', 'targets': {'smithi067.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHc368b/ld8CXeRgZD2wZKDabNI1eXfuXEMNsxKrM0ylBNhZkR2Ojg21ljXQq705FBt6xWiWXgxuXG2KNvfnxwY=', 'smithi082.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOSfXZVLelftUITjlRAgf/G/9OTlf8HfMEZz+qIw9jTkkEqgB1v5m3QDgDoN4xfB9xrvptDpktppp2QW52tuIEI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 2, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': '*', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'create_rbd_pool': False, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'sha1': '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc', 'cluster': 'ceph'}}, {'thrashosds': {'chance_test_min_size': 3, 'timeout': 1200}}, {'rados': {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_use_overwrites': True, 'objects': 50, 'op_weights': {'copy_from': 50, 'delete': 50, 'read': 100, 'rollback': 50, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-30_00:41:04', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-30T12:54:41.626 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-30T12:54:41.657 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-30T12:54:41.663 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/run/ceph 2024-03-30T12:54:41.705 DEBUG:teuthology.orchestra.run.smithi082:> sudo install -d -m0777 -- /var/run/ceph 2024-03-30T12:54:41.717 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-30T12:54:41.717 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2024-03-30T12:54:41.764 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-30T12:54:41.764 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_1 2024-03-30T12:54:41.812 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-30T12:54:41.813 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T12:54:41.813 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 870 Links: 1 2024-03-30T12:54:41.813 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T12:54:41.813 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-03-30 12:49:14.536548659 +0000 2024-03-30T12:54:41.813 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-03-30 12:49:14.288550871 +0000 2024-03-30T12:54:41.813 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-03-30 12:49:14.288550871 +0000 2024-03-30T12:54:41.813 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2024-03-30T12:54:41.813 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-30T12:54:41.867 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-03-30T12:54:41.867 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-03-30T12:54:41.867 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000374508 s, 1.4 MB/s 2024-03-30T12:54:41.869 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-30T12:54:41.918 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_2 2024-03-30T12:54:41.964 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-30T12:54:41.964 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T12:54:41.964 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 888 Links: 1 2024-03-30T12:54:41.964 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T12:54:41.964 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-03-30 12:49:15.048544092 +0000 2024-03-30T12:54:41.964 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-03-30 12:49:14.804546269 +0000 2024-03-30T12:54:41.964 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-03-30 12:49:14.804546269 +0000 2024-03-30T12:54:41.965 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2024-03-30T12:54:41.965 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-30T12:54:42.019 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-03-30T12:54:42.019 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-03-30T12:54:42.019 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000351827 s, 1.5 MB/s 2024-03-30T12:54:42.020 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-30T12:54:42.069 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_3 2024-03-30T12:54:42.123 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-30T12:54:42.123 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T12:54:42.123 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 906 Links: 1 2024-03-30T12:54:42.123 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T12:54:42.123 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-03-30 12:49:15.300541844 +0000 2024-03-30T12:54:42.123 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-03-30 12:49:15.296541880 +0000 2024-03-30T12:54:42.123 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-03-30 12:49:15.296541880 +0000 2024-03-30T12:54:42.123 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2024-03-30T12:54:42.123 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-30T12:54:42.179 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-03-30T12:54:42.179 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-03-30T12:54:42.179 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000341234 s, 1.5 MB/s 2024-03-30T12:54:42.180 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-30T12:54:42.230 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_4 2024-03-30T12:54:42.276 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-30T12:54:42.276 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T12:54:42.276 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 920 Links: 1 2024-03-30T12:54:42.276 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T12:54:42.277 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-03-30 12:49:16.032535314 +0000 2024-03-30T12:54:42.277 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-03-30 12:49:15.784537526 +0000 2024-03-30T12:54:42.277 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-03-30 12:49:15.784537526 +0000 2024-03-30T12:54:42.277 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2024-03-30T12:54:42.277 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-30T12:54:42.331 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-03-30T12:54:42.331 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-03-30T12:54:42.331 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.00036775 s, 1.4 MB/s 2024-03-30T12:54:42.332 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-30T12:54:42.381 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-30T12:54:42.381 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-30T12:54:42.382 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/scratch_devs of=/dev/stdout 2024-03-30T12:54:42.387 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-30T12:54:42.387 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_1 2024-03-30T12:54:42.433 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-30T12:54:42.434 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T12:54:42.434 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 884 Links: 1 2024-03-30T12:54:42.434 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T12:54:42.434 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-03-30 12:49:09.501221769 +0000 2024-03-30T12:54:42.434 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-03-30 12:49:09.261223911 +0000 2024-03-30T12:54:42.434 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-03-30 12:49:09.261223911 +0000 2024-03-30T12:54:42.434 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: - 2024-03-30T12:54:42.434 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-30T12:54:42.487 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-03-30T12:54:42.487 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-03-30T12:54:42.487 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000406796 s, 1.3 MB/s 2024-03-30T12:54:42.489 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-30T12:54:42.540 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_2 2024-03-30T12:54:42.588 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-30T12:54:42.588 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T12:54:42.589 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 895 Links: 1 2024-03-30T12:54:42.589 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T12:54:42.589 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-03-30 12:49:09.977217523 +0000 2024-03-30T12:54:42.589 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-03-30 12:49:09.729219736 +0000 2024-03-30T12:54:42.589 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-03-30 12:49:09.729219736 +0000 2024-03-30T12:54:42.589 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: - 2024-03-30T12:54:42.589 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-30T12:54:42.642 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-03-30T12:54:42.643 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-03-30T12:54:42.643 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000419034 s, 1.2 MB/s 2024-03-30T12:54:42.644 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-30T12:54:42.694 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_3 2024-03-30T12:54:42.742 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-30T12:54:42.742 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T12:54:42.742 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 917 Links: 1 2024-03-30T12:54:42.742 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T12:54:42.743 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-03-30 12:49:10.213215418 +0000 2024-03-30T12:54:42.743 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-03-30 12:49:10.209215453 +0000 2024-03-30T12:54:42.743 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-03-30 12:49:10.209215453 +0000 2024-03-30T12:54:42.743 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: - 2024-03-30T12:54:42.743 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-30T12:54:42.795 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-03-30T12:54:42.796 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-03-30T12:54:42.796 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.00045233 s, 1.1 MB/s 2024-03-30T12:54:42.797 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-30T12:54:42.846 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_4 2024-03-30T12:54:42.893 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-30T12:54:42.893 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-30T12:54:42.893 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 937 Links: 1 2024-03-30T12:54:42.893 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-30T12:54:42.893 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-03-30 12:49:10.993208459 +0000 2024-03-30T12:54:42.893 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-03-30 12:49:10.725210850 +0000 2024-03-30T12:54:42.893 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-03-30 12:49:10.725210850 +0000 2024-03-30T12:54:42.894 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: - 2024-03-30T12:54:42.894 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-30T12:54:42.948 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-03-30T12:54:42.948 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-03-30T12:54:42.948 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.00029779 s, 1.7 MB/s 2024-03-30T12:54:42.950 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-30T12:54:42.998 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-30T12:54:42.998 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi082.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-30T12:54:42.998 INFO:tasks.ceph:Generating config... 2024-03-30T12:54:42.999 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-30T12:54:42.999 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-30T12:54:43.000 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-30T12:54:43.000 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-03-30T12:54:43.000 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-30T12:54:43.000 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-30T12:54:43.000 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-30T12:54:43.000 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-30T12:54:43.000 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-30T12:54:43.000 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-30T12:54:43.000 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-30T12:54:43.000 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-30T12:54:43.000 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-30T12:54:43.000 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-30T12:54:43.001 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-30T12:54:43.001 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-30T12:54:43.001 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2024-03-30T12:54:43.001 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-30T12:54:43.001 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-30T12:54:43.001 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-30T12:54:43.001 INFO:tasks.ceph:[osd] osd max backfills = 2 2024-03-30T12:54:43.001 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-30T12:54:43.001 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-30T12:54:43.001 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-30T12:54:43.001 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-30T12:54:43.001 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-30T12:54:43.001 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-30T12:54:43.001 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-03-30T12:54:43.001 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-03-30T12:54:43.001 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-30T12:54:43.002 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = * 2024-03-30T12:54:43.002 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-30T12:54:43.002 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-30T12:54:43.002 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-03-30T12:54:43.002 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-03-30T12:54:43.002 INFO:tasks.ceph:[global] ms inject delay type = osd 2024-03-30T12:54:43.002 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-03-30T12:54:43.002 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-03-30T12:54:43.002 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-30T12:54:43.002 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-30T12:54:43.002 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-30T12:54:43.002 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-30T12:54:43.002 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-30T12:54:43.002 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-30T12:54:43.003 INFO:tasks.ceph:Setting up mon.a... 2024-03-30T12:54:43.003 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-30T12:54:43.038 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.keyring 2024-03-30T12:54:43.053 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-30T12:54:43.131 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-30T12:54:43.188 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.67'), ('mon.c', '[v2:172.21.15.67:3301,v1:172.21.15.67:6790]'), ('mon.b', '172.21.15.82')] 2024-03-30T12:54:43.188 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.67,[v2:172.21.15.67:3301,v1:172.21.15.67:6790],172.21.15.82', 'enable experimental unrecoverable data corrupting features': '*', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 2, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-30T12:54:43.189 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-30T12:54:43.189 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-30T12:54:43.236 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.67 --addv c '[v2:172.21.15.67:3301,v1:172.21.15.67:6790]' --add b 172.21.15.82 --print /home/ubuntu/cephtest/ceph.monmap 2024-03-30T12:54:43.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:43.326+0000 7f0fbd325100 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:54:43.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:43.326+0000 7f0fbd325100 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:54:43.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:43.326+0000 7f0fbd325100 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:54:43.325 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-30T12:54:43.325 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: generated fsid 0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a 2024-03-30T12:54:43.325 INFO:teuthology.orchestra.run.smithi067.stdout:setting min_mon_release = quincy 2024-03-30T12:54:43.325 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 0 2024-03-30T12:54:43.325 INFO:teuthology.orchestra.run.smithi067.stdout:fsid 0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a 2024-03-30T12:54:43.325 INFO:teuthology.orchestra.run.smithi067.stdout:last_changed 2024-03-30T12:54:43.327476+0000 2024-03-30T12:54:43.325 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-03-30T12:54:43.327476+0000 2024-03-30T12:54:43.325 INFO:teuthology.orchestra.run.smithi067.stdout:min_mon_release 17 (quincy) 2024-03-30T12:54:43.325 INFO:teuthology.orchestra.run.smithi067.stdout:election_strategy: 3 2024-03-30T12:54:43.325 INFO:teuthology.orchestra.run.smithi067.stdout:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-03-30T12:54:43.325 INFO:teuthology.orchestra.run.smithi067.stdout:1: [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] mon.b 2024-03-30T12:54:43.325 INFO:teuthology.orchestra.run.smithi067.stdout:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2024-03-30T12:54:43.325 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-30T12:54:43.327 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-30T12:54:43.331 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a... 2024-03-30T12:54:43.332 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-30T12:54:43.373 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-30T12:54:43.398 INFO:teuthology.orchestra.run.smithi067.stdout:[global] 2024-03-30T12:54:43.399 INFO:teuthology.orchestra.run.smithi067.stdout: chdir = "" 2024-03-30T12:54:43.399 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-30T12:54:43.399 INFO:teuthology.orchestra.run.smithi067.stdout: auth supported = cephx 2024-03-30T12:54:43.399 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.399 INFO:teuthology.orchestra.run.smithi067.stdout: filestore xattr use omap = true 2024-03-30T12:54:43.399 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.399 INFO:teuthology.orchestra.run.smithi067.stdout: mon clock drift allowed = 1.000 2024-03-30T12:54:43.399 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.399 INFO:teuthology.orchestra.run.smithi067.stdout: osd crush chooseleaf type = 0 2024-03-30T12:54:43.399 INFO:teuthology.orchestra.run.smithi067.stdout: auth debug = true 2024-03-30T12:54:43.399 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.399 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on old message = true 2024-03-30T12:54:43.400 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on bug = true 2024-03-30T12:54:43.400 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.400 INFO:teuthology.orchestra.run.smithi067.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-30T12:54:43.400 INFO:teuthology.orchestra.run.smithi067.stdout: mon pg warn max object skew = 0 2024-03-30T12:54:43.400 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.400 INFO:teuthology.orchestra.run.smithi067.stdout: # disable pg_autoscaler by default for new pools 2024-03-30T12:54:43.400 INFO:teuthology.orchestra.run.smithi067.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-30T12:54:43.400 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.400 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default size = 2 2024-03-30T12:54:43.400 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.400 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow primary affinity = true 2024-03-30T12:54:43.400 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow pg remap = true 2024-03-30T12:54:43.400 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on legacy crush tunables = false 2024-03-30T12:54:43.400 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on crush straw calc version zero = false 2024-03-30T12:54:43.401 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on no sortbitwise = false 2024-03-30T12:54:43.401 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on osd down out interval zero = false 2024-03-30T12:54:43.401 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on too few osds = false 2024-03-30T12:54:43.401 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-30T12:54:43.401 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-30T12:54:43.401 INFO:teuthology.orchestra.run.smithi067.stdout: mon_allow_pool_size_one = true 2024-03-30T12:54:43.401 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.401 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-30T12:54:43.401 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.401 INFO:teuthology.orchestra.run.smithi067.stdout: osd default data pool replay window = 5 2024-03-30T12:54:43.401 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.401 INFO:teuthology.orchestra.run.smithi067.stdout: mon allow pool delete = true 2024-03-30T12:54:43.401 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.401 INFO:teuthology.orchestra.run.smithi067.stdout: mon cluster log file level = debug 2024-03-30T12:54:43.401 INFO:teuthology.orchestra.run.smithi067.stdout: debug asserts on shutdown = true 2024-03-30T12:54:43.402 INFO:teuthology.orchestra.run.smithi067.stdout: mon health detail to clog = false 2024-03-30T12:54:43.402 INFO:teuthology.orchestra.run.smithi067.stdout: mon host = "172.21.15.67,[v2:172.21.15.67:3301,v1:172.21.15.67:6790],172.21.15.82" 2024-03-30T12:54:43.402 INFO:teuthology.orchestra.run.smithi067.stdout: enable experimental unrecoverable data corrupting features = * 2024-03-30T12:54:43.402 INFO:teuthology.orchestra.run.smithi067.stdout: mon client directed command retry = 5 2024-03-30T12:54:43.402 INFO:teuthology.orchestra.run.smithi067.stdout: mon election default strategy = 3 2024-03-30T12:54:43.402 INFO:teuthology.orchestra.run.smithi067.stdout: ms inject delay max = 1 2024-03-30T12:54:43.402 INFO:teuthology.orchestra.run.smithi067.stdout: ms inject delay probability = 0.005 2024-03-30T12:54:43.402 INFO:teuthology.orchestra.run.smithi067.stdout: ms inject delay type = osd 2024-03-30T12:54:43.402 INFO:teuthology.orchestra.run.smithi067.stdout: ms inject internal delays = 0.002 2024-03-30T12:54:43.402 INFO:teuthology.orchestra.run.smithi067.stdout: ms inject socket failures = 2500 2024-03-30T12:54:43.402 INFO:teuthology.orchestra.run.smithi067.stdout: fsid = 0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a 2024-03-30T12:54:43.402 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.403 INFO:teuthology.orchestra.run.smithi067.stdout:[osd] 2024-03-30T12:54:43.403 INFO:teuthology.orchestra.run.smithi067.stdout: osd journal size = 100 2024-03-30T12:54:43.403 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.403 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub load threshold = 5.0 2024-03-30T12:54:43.403 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub max interval = 120 2024-03-30T12:54:43.403 INFO:teuthology.orchestra.run.smithi067.stdout: osd mclock profile = high_recovery_ops 2024-03-30T12:54:43.403 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.403 INFO:teuthology.orchestra.run.smithi067.stdout: osd recover clone overlap = true 2024-03-30T12:54:43.404 INFO:teuthology.orchestra.run.smithi067.stdout: osd recovery max chunk = 1048576 2024-03-30T12:54:43.404 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.404 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug shutdown = true 2024-03-30T12:54:43.404 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug op order = true 2024-03-30T12:54:43.404 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify stray on activate = true 2024-03-30T12:54:43.404 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug trim objects = true 2024-03-30T12:54:43.404 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.404 INFO:teuthology.orchestra.run.smithi067.stdout: osd open classes on start = true 2024-03-30T12:54:43.404 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug pg log writeout = true 2024-03-30T12:54:43.404 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.404 INFO:teuthology.orchestra.run.smithi067.stdout: osd deep scrub update digest min age = 30 2024-03-30T12:54:43.404 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.404 INFO:teuthology.orchestra.run.smithi067.stdout: osd map max advance = 10 2024-03-30T12:54:43.404 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.404 INFO:teuthology.orchestra.run.smithi067.stdout: journal zero on create = true 2024-03-30T12:54:43.405 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.405 INFO:teuthology.orchestra.run.smithi067.stdout: filestore ondisk finisher threads = 3 2024-03-30T12:54:43.405 INFO:teuthology.orchestra.run.smithi067.stdout: filestore apply finisher threads = 3 2024-03-30T12:54:43.405 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.405 INFO:teuthology.orchestra.run.smithi067.stdout: bdev debug aio = true 2024-03-30T12:54:43.405 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug misdirected ops = true 2024-03-30T12:54:43.405 INFO:teuthology.orchestra.run.smithi067.stdout: debug monc = 20 2024-03-30T12:54:43.405 INFO:teuthology.orchestra.run.smithi067.stdout: bdev async discard = True 2024-03-30T12:54:43.405 INFO:teuthology.orchestra.run.smithi067.stdout: bdev enable discard = True 2024-03-30T12:54:43.405 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore allocator = bitmap 2024-03-30T12:54:43.405 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore block size = 96636764160 2024-03-30T12:54:43.405 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore fsck on mount = True 2024-03-30T12:54:43.405 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore zero block detection = True 2024-03-30T12:54:43.405 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluefs = 20 2024-03-30T12:54:43.406 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluestore = 20 2024-03-30T12:54:43.406 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-30T12:54:43.406 INFO:teuthology.orchestra.run.smithi067.stdout: debug osd = 20 2024-03-30T12:54:43.406 INFO:teuthology.orchestra.run.smithi067.stdout: debug rocksdb = 10 2024-03-30T12:54:43.406 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd backfillfull_ratio = 0.85 2024-03-30T12:54:43.406 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd full ratio = 0.9 2024-03-30T12:54:43.406 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd nearfull ratio = 0.8 2024-03-30T12:54:43.406 INFO:teuthology.orchestra.run.smithi067.stdout: osd blocked scrub grace period = 3600 2024-03-30T12:54:43.406 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug reject backfill probability = 0.3 2024-03-30T12:54:43.406 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify cached snaps = True 2024-03-30T12:54:43.406 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify missing on start = True 2024-03-30T12:54:43.406 INFO:teuthology.orchestra.run.smithi067.stdout: osd failsafe full ratio = 0.95 2024-03-30T12:54:43.406 INFO:teuthology.orchestra.run.smithi067.stdout: osd max backfills = 2 2024-03-30T12:54:43.406 INFO:teuthology.orchestra.run.smithi067.stdout: osd max markdown count = 1000 2024-03-30T12:54:43.406 INFO:teuthology.orchestra.run.smithi067.stdout: osd mclock override recovery settings = True 2024-03-30T12:54:43.407 INFO:teuthology.orchestra.run.smithi067.stdout: osd objectstore = bluestore 2024-03-30T12:54:43.407 INFO:teuthology.orchestra.run.smithi067.stdout: osd op queue = debug_random 2024-03-30T12:54:43.407 INFO:teuthology.orchestra.run.smithi067.stdout: osd op queue cut off = debug_random 2024-03-30T12:54:43.407 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub min interval = 60 2024-03-30T12:54:43.407 INFO:teuthology.orchestra.run.smithi067.stdout: osd shutdown pgref assert = True 2024-03-30T12:54:43.407 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.407 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr] 2024-03-30T12:54:43.407 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-30T12:54:43.407 INFO:teuthology.orchestra.run.smithi067.stdout: debug mgr = 20 2024-03-30T12:54:43.407 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-03-30T12:54:43.407 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-03-30T12:54:43.407 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-03-30T12:54:43.407 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-03-30T12:54:43.407 INFO:teuthology.orchestra.run.smithi067.stdout: mgr/telemetry/nag = false 2024-03-30T12:54:43.407 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.408 INFO:teuthology.orchestra.run.smithi067.stdout:[mon] 2024-03-30T12:54:43.408 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-03-30T12:54:43.408 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-03-30T12:54:43.408 INFO:teuthology.orchestra.run.smithi067.stdout: debug paxos = 20 2024-03-30T12:54:43.408 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-03-30T12:54:43.408 INFO:teuthology.orchestra.run.smithi067.stdout: mon data avail warn = 5 2024-03-30T12:54:43.408 INFO:teuthology.orchestra.run.smithi067.stdout: mon mgr mkfs grace = 240 2024-03-30T12:54:43.408 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-03-30T12:54:43.408 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd reporter subtree level = osd 2024-03-30T12:54:43.408 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd prime pg temp = true 2024-03-30T12:54:43.408 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-03-30T12:54:43.408 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.408 INFO:teuthology.orchestra.run.smithi067.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-30T12:54:43.408 INFO:teuthology.orchestra.run.smithi067.stdout: auth mon ticket ttl = 660 # 11m 2024-03-30T12:54:43.408 INFO:teuthology.orchestra.run.smithi067.stdout: auth service ticket ttl = 240 # 4m 2024-03-30T12:54:43.409 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.409 INFO:teuthology.orchestra.run.smithi067.stdout: # don't complain about insecure global_id in the test suite 2024-03-30T12:54:43.409 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-30T12:54:43.409 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-30T12:54:43.409 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.409 INFO:teuthology.orchestra.run.smithi067.stdout: # 1m isn't quite enough 2024-03-30T12:54:43.409 INFO:teuthology.orchestra.run.smithi067.stdout: mon_down_mkfs_grace = 2m 2024-03-30T12:54:43.409 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.409 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_filestore_osds = false 2024-03-30T12:54:43.409 INFO:teuthology.orchestra.run.smithi067.stdout: mon scrub interval = 300 2024-03-30T12:54:43.409 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:54:43.409 INFO:teuthology.orchestra.run.smithi067.stdout:[client] 2024-03-30T12:54:43.409 INFO:teuthology.orchestra.run.smithi067.stdout: rgw cache enabled = true 2024-03-30T12:54:43.409 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable ops log = true 2024-03-30T12:54:43.409 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable usage log = true 2024-03-30T12:54:43.410 INFO:teuthology.orchestra.run.smithi067.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-30T12:54:43.410 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-30T12:54:43.410 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.a] 2024-03-30T12:54:43.410 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.c] 2024-03-30T12:54:43.410 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.b] 2024-03-30T12:54:43.411 INFO:teuthology.orchestra.run.smithi082.stdout:[global] 2024-03-30T12:54:43.411 INFO:teuthology.orchestra.run.smithi082.stdout: chdir = "" 2024-03-30T12:54:43.411 INFO:teuthology.orchestra.run.smithi082.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-30T12:54:43.411 INFO:teuthology.orchestra.run.smithi082.stdout: auth supported = cephx 2024-03-30T12:54:43.411 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.411 INFO:teuthology.orchestra.run.smithi082.stdout: filestore xattr use omap = true 2024-03-30T12:54:43.411 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.411 INFO:teuthology.orchestra.run.smithi082.stdout: mon clock drift allowed = 1.000 2024-03-30T12:54:43.411 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.411 INFO:teuthology.orchestra.run.smithi082.stdout: osd crush chooseleaf type = 0 2024-03-30T12:54:43.411 INFO:teuthology.orchestra.run.smithi082.stdout: auth debug = true 2024-03-30T12:54:43.412 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.412 INFO:teuthology.orchestra.run.smithi082.stdout: ms die on old message = true 2024-03-30T12:54:43.412 INFO:teuthology.orchestra.run.smithi082.stdout: ms die on bug = true 2024-03-30T12:54:43.412 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.412 INFO:teuthology.orchestra.run.smithi082.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-30T12:54:43.412 INFO:teuthology.orchestra.run.smithi082.stdout: mon pg warn max object skew = 0 2024-03-30T12:54:43.412 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.412 INFO:teuthology.orchestra.run.smithi082.stdout: # disable pg_autoscaler by default for new pools 2024-03-30T12:54:43.412 INFO:teuthology.orchestra.run.smithi082.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-30T12:54:43.412 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.412 INFO:teuthology.orchestra.run.smithi082.stdout: osd pool default size = 2 2024-03-30T12:54:43.412 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.412 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd allow primary affinity = true 2024-03-30T12:54:43.412 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd allow pg remap = true 2024-03-30T12:54:43.412 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on legacy crush tunables = false 2024-03-30T12:54:43.413 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on crush straw calc version zero = false 2024-03-30T12:54:43.413 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on no sortbitwise = false 2024-03-30T12:54:43.413 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on osd down out interval zero = false 2024-03-30T12:54:43.413 INFO:teuthology.orchestra.run.smithi082.stdout: mon warn on too few osds = false 2024-03-30T12:54:43.413 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-30T12:54:43.413 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-30T12:54:43.413 INFO:teuthology.orchestra.run.smithi082.stdout: mon_allow_pool_size_one = true 2024-03-30T12:54:43.413 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.413 INFO:teuthology.orchestra.run.smithi082.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-30T12:54:43.414 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.414 INFO:teuthology.orchestra.run.smithi082.stdout: osd default data pool replay window = 5 2024-03-30T12:54:43.414 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.414 INFO:teuthology.orchestra.run.smithi082.stdout: mon allow pool delete = true 2024-03-30T12:54:43.414 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.414 INFO:teuthology.orchestra.run.smithi082.stdout: mon cluster log file level = debug 2024-03-30T12:54:43.414 INFO:teuthology.orchestra.run.smithi082.stdout: debug asserts on shutdown = true 2024-03-30T12:54:43.414 INFO:teuthology.orchestra.run.smithi082.stdout: mon health detail to clog = false 2024-03-30T12:54:43.414 INFO:teuthology.orchestra.run.smithi082.stdout: mon host = "172.21.15.67,[v2:172.21.15.67:3301,v1:172.21.15.67:6790],172.21.15.82" 2024-03-30T12:54:43.414 INFO:teuthology.orchestra.run.smithi082.stdout: enable experimental unrecoverable data corrupting features = * 2024-03-30T12:54:43.414 INFO:teuthology.orchestra.run.smithi082.stdout: mon client directed command retry = 5 2024-03-30T12:54:43.414 INFO:teuthology.orchestra.run.smithi082.stdout: mon election default strategy = 3 2024-03-30T12:54:43.414 INFO:teuthology.orchestra.run.smithi082.stdout: ms inject delay max = 1 2024-03-30T12:54:43.414 INFO:teuthology.orchestra.run.smithi082.stdout: ms inject delay probability = 0.005 2024-03-30T12:54:43.415 INFO:teuthology.orchestra.run.smithi082.stdout: ms inject delay type = osd 2024-03-30T12:54:43.415 INFO:teuthology.orchestra.run.smithi082.stdout: ms inject internal delays = 0.002 2024-03-30T12:54:43.415 INFO:teuthology.orchestra.run.smithi082.stdout: ms inject socket failures = 2500 2024-03-30T12:54:43.415 INFO:teuthology.orchestra.run.smithi082.stdout: fsid = 0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a 2024-03-30T12:54:43.415 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.415 INFO:teuthology.orchestra.run.smithi082.stdout:[osd] 2024-03-30T12:54:43.415 INFO:teuthology.orchestra.run.smithi082.stdout: osd journal size = 100 2024-03-30T12:54:43.415 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.415 INFO:teuthology.orchestra.run.smithi082.stdout: osd scrub load threshold = 5.0 2024-03-30T12:54:43.415 INFO:teuthology.orchestra.run.smithi082.stdout: osd scrub max interval = 120 2024-03-30T12:54:43.415 INFO:teuthology.orchestra.run.smithi082.stdout: osd mclock profile = high_recovery_ops 2024-03-30T12:54:43.415 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.415 INFO:teuthology.orchestra.run.smithi082.stdout: osd recover clone overlap = true 2024-03-30T12:54:43.415 INFO:teuthology.orchestra.run.smithi082.stdout: osd recovery max chunk = 1048576 2024-03-30T12:54:43.415 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.416 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug shutdown = true 2024-03-30T12:54:43.416 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug op order = true 2024-03-30T12:54:43.416 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug verify stray on activate = true 2024-03-30T12:54:43.416 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug trim objects = true 2024-03-30T12:54:43.416 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.416 INFO:teuthology.orchestra.run.smithi082.stdout: osd open classes on start = true 2024-03-30T12:54:43.416 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug pg log writeout = true 2024-03-30T12:54:43.416 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.416 INFO:teuthology.orchestra.run.smithi082.stdout: osd deep scrub update digest min age = 30 2024-03-30T12:54:43.416 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.416 INFO:teuthology.orchestra.run.smithi082.stdout: osd map max advance = 10 2024-03-30T12:54:43.416 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.416 INFO:teuthology.orchestra.run.smithi082.stdout: journal zero on create = true 2024-03-30T12:54:43.416 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.416 INFO:teuthology.orchestra.run.smithi082.stdout: filestore ondisk finisher threads = 3 2024-03-30T12:54:43.417 INFO:teuthology.orchestra.run.smithi082.stdout: filestore apply finisher threads = 3 2024-03-30T12:54:43.417 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.417 INFO:teuthology.orchestra.run.smithi082.stdout: bdev debug aio = true 2024-03-30T12:54:43.417 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug misdirected ops = true 2024-03-30T12:54:43.417 INFO:teuthology.orchestra.run.smithi082.stdout: debug monc = 20 2024-03-30T12:54:43.417 INFO:teuthology.orchestra.run.smithi082.stdout: bdev async discard = True 2024-03-30T12:54:43.417 INFO:teuthology.orchestra.run.smithi082.stdout: bdev enable discard = True 2024-03-30T12:54:43.417 INFO:teuthology.orchestra.run.smithi082.stdout: bluestore allocator = bitmap 2024-03-30T12:54:43.417 INFO:teuthology.orchestra.run.smithi082.stdout: bluestore block size = 96636764160 2024-03-30T12:54:43.417 INFO:teuthology.orchestra.run.smithi082.stdout: bluestore fsck on mount = True 2024-03-30T12:54:43.417 INFO:teuthology.orchestra.run.smithi082.stdout: bluestore zero block detection = True 2024-03-30T12:54:43.417 INFO:teuthology.orchestra.run.smithi082.stdout: debug bluefs = 20 2024-03-30T12:54:43.417 INFO:teuthology.orchestra.run.smithi082.stdout: debug bluestore = 20 2024-03-30T12:54:43.417 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2024-03-30T12:54:43.418 INFO:teuthology.orchestra.run.smithi082.stdout: debug osd = 20 2024-03-30T12:54:43.418 INFO:teuthology.orchestra.run.smithi082.stdout: debug rocksdb = 10 2024-03-30T12:54:43.418 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd backfillfull_ratio = 0.85 2024-03-30T12:54:43.418 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd full ratio = 0.9 2024-03-30T12:54:43.418 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd nearfull ratio = 0.8 2024-03-30T12:54:43.418 INFO:teuthology.orchestra.run.smithi082.stdout: osd blocked scrub grace period = 3600 2024-03-30T12:54:43.418 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug reject backfill probability = 0.3 2024-03-30T12:54:43.418 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug verify cached snaps = True 2024-03-30T12:54:43.418 INFO:teuthology.orchestra.run.smithi082.stdout: osd debug verify missing on start = True 2024-03-30T12:54:43.418 INFO:teuthology.orchestra.run.smithi082.stdout: osd failsafe full ratio = 0.95 2024-03-30T12:54:43.418 INFO:teuthology.orchestra.run.smithi082.stdout: osd max backfills = 2 2024-03-30T12:54:43.418 INFO:teuthology.orchestra.run.smithi082.stdout: osd max markdown count = 1000 2024-03-30T12:54:43.418 INFO:teuthology.orchestra.run.smithi082.stdout: osd mclock override recovery settings = True 2024-03-30T12:54:43.418 INFO:teuthology.orchestra.run.smithi082.stdout: osd objectstore = bluestore 2024-03-30T12:54:43.418 INFO:teuthology.orchestra.run.smithi082.stdout: osd op queue = debug_random 2024-03-30T12:54:43.419 INFO:teuthology.orchestra.run.smithi082.stdout: osd op queue cut off = debug_random 2024-03-30T12:54:43.419 INFO:teuthology.orchestra.run.smithi082.stdout: osd scrub min interval = 60 2024-03-30T12:54:43.419 INFO:teuthology.orchestra.run.smithi082.stdout: osd shutdown pgref assert = True 2024-03-30T12:54:43.419 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.419 INFO:teuthology.orchestra.run.smithi082.stdout:[mgr] 2024-03-30T12:54:43.419 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2024-03-30T12:54:43.419 INFO:teuthology.orchestra.run.smithi082.stdout: debug mgr = 20 2024-03-30T12:54:43.419 INFO:teuthology.orchestra.run.smithi082.stdout: debug mon = 20 2024-03-30T12:54:43.419 INFO:teuthology.orchestra.run.smithi082.stdout: debug auth = 20 2024-03-30T12:54:43.419 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min pgs per osd = 4 2024-03-30T12:54:43.419 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min bytes per osd = 10 2024-03-30T12:54:43.419 INFO:teuthology.orchestra.run.smithi082.stdout: mgr/telemetry/nag = false 2024-03-30T12:54:43.419 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.419 INFO:teuthology.orchestra.run.smithi082.stdout:[mon] 2024-03-30T12:54:43.419 INFO:teuthology.orchestra.run.smithi082.stdout: debug ms = 1 2024-03-30T12:54:43.420 INFO:teuthology.orchestra.run.smithi082.stdout: debug mon = 20 2024-03-30T12:54:43.420 INFO:teuthology.orchestra.run.smithi082.stdout: debug paxos = 20 2024-03-30T12:54:43.420 INFO:teuthology.orchestra.run.smithi082.stdout: debug auth = 20 2024-03-30T12:54:43.420 INFO:teuthology.orchestra.run.smithi082.stdout: mon data avail warn = 5 2024-03-30T12:54:43.420 INFO:teuthology.orchestra.run.smithi082.stdout: mon mgr mkfs grace = 240 2024-03-30T12:54:43.420 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min pgs per osd = 4 2024-03-30T12:54:43.420 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd reporter subtree level = osd 2024-03-30T12:54:43.420 INFO:teuthology.orchestra.run.smithi082.stdout: mon osd prime pg temp = true 2024-03-30T12:54:43.420 INFO:teuthology.orchestra.run.smithi082.stdout: mon reweight min bytes per osd = 10 2024-03-30T12:54:43.420 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.420 INFO:teuthology.orchestra.run.smithi082.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-30T12:54:43.420 INFO:teuthology.orchestra.run.smithi082.stdout: auth mon ticket ttl = 660 # 11m 2024-03-30T12:54:43.420 INFO:teuthology.orchestra.run.smithi082.stdout: auth service ticket ttl = 240 # 4m 2024-03-30T12:54:43.420 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.420 INFO:teuthology.orchestra.run.smithi082.stdout: # don't complain about insecure global_id in the test suite 2024-03-30T12:54:43.420 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-30T12:54:43.421 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-30T12:54:43.421 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.421 INFO:teuthology.orchestra.run.smithi082.stdout: # 1m isn't quite enough 2024-03-30T12:54:43.421 INFO:teuthology.orchestra.run.smithi082.stdout: mon_down_mkfs_grace = 2m 2024-03-30T12:54:43.421 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.421 INFO:teuthology.orchestra.run.smithi082.stdout: mon_warn_on_filestore_osds = false 2024-03-30T12:54:43.421 INFO:teuthology.orchestra.run.smithi082.stdout: mon scrub interval = 300 2024-03-30T12:54:43.421 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-03-30T12:54:43.421 INFO:teuthology.orchestra.run.smithi082.stdout:[client] 2024-03-30T12:54:43.421 INFO:teuthology.orchestra.run.smithi082.stdout: rgw cache enabled = true 2024-03-30T12:54:43.421 INFO:teuthology.orchestra.run.smithi082.stdout: rgw enable ops log = true 2024-03-30T12:54:43.421 INFO:teuthology.orchestra.run.smithi082.stdout: rgw enable usage log = true 2024-03-30T12:54:43.421 INFO:teuthology.orchestra.run.smithi082.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-30T12:54:43.421 INFO:teuthology.orchestra.run.smithi082.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-30T12:54:43.421 INFO:teuthology.orchestra.run.smithi082.stdout:[mon.a] 2024-03-30T12:54:43.422 INFO:teuthology.orchestra.run.smithi082.stdout:[mon.c] 2024-03-30T12:54:43.422 INFO:teuthology.orchestra.run.smithi082.stdout:[mon.b] 2024-03-30T12:54:43.423 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-30T12:54:43.423 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-30T12:54:43.511 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-30T12:54:43.511 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-30T12:54:43.511 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-30T12:54:43.517 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-30T12:54:43.517 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-30T12:54:43.563 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi067.front.sepia.ceph.com 2024-03-30T12:54:43.563 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-30T12:54:43.564 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-30T12:54:43.564 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-30T12:54:43.628 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-30T12:54:43.628 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-30T12:54:43.676 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi082.front.sepia.ceph.com 2024-03-30T12:54:43.676 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-30T12:54:43.676 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-30T12:54:43.677 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-30T12:54:43.700 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-30T12:54:43.700 DEBUG:teuthology.orchestra.run.smithi082:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-30T12:54:43.749 INFO:tasks.ceph:Setting up mon nodes... 2024-03-30T12:54:43.749 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-30T12:54:43.749 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-30T12:54:43.794 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-30T12:54:43.797 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-30T12:54:43.841 INFO:teuthology.orchestra.run.smithi082.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-30T12:54:43.871 INFO:tasks.ceph:Setting up mds nodes... 2024-03-30T12:54:43.871 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-30T12:54:43.871 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-30T12:54:43.909 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-30T12:54:43.920 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-30T12:54:43.957 INFO:teuthology.orchestra.run.smithi082.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-30T12:54:43.968 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-30T12:54:43.968 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi082.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-30T12:54:43.968 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-30T12:54:43.980 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-30T12:54:43.981 INFO:tasks.ceph:role: osd.0 2024-03-30T12:54:43.981 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com 2024-03-30T12:54:43.981 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-30T12:54:44.044 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-30T12:54:44.044 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-30T12:54:44.044 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-30T12:54:44.044 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-30T12:54:44.045 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-30T12:54:44.045 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-03-30T12:54:44.045 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-30T12:54:44.045 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-30T12:54:44.045 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-30T12:54:44.045 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-30T12:54:44.506 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-03-30T12:54:44.509 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-03-30T12:54:44.509 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-30T12:54:44.604 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-30T12:54:44.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T12:54:44.615 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: /sbin/restorecon: command not found 2024-03-30T12:54:44.615 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-30T12:54:44.666 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-30T12:54:44.666 INFO:tasks.ceph:role: osd.1 2024-03-30T12:54:44.666 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com 2024-03-30T12:54:44.666 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-30T12:54:44.730 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-30T12:54:44.730 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-30T12:54:44.730 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-30T12:54:44.730 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-30T12:54:44.730 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-30T12:54:44.730 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-03-30T12:54:44.730 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-30T12:54:44.730 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-30T12:54:44.730 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-30T12:54:44.730 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-30T12:54:45.197 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-03-30T12:54:45.199 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-03-30T12:54:45.199 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-30T12:54:45.335 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-30T12:54:45.347 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: /sbin/restorecon: command not found 2024-03-30T12:54:45.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T12:54:45.348 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-30T12:54:45.406 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-30T12:54:45.406 INFO:tasks.ceph:role: osd.2 2024-03-30T12:54:45.406 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com 2024-03-30T12:54:45.406 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-30T12:54:45.468 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-30T12:54:45.468 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-30T12:54:45.468 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-30T12:54:45.468 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-30T12:54:45.468 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-30T12:54:45.468 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-03-30T12:54:45.468 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-30T12:54:45.469 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-30T12:54:45.469 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-30T12:54:45.469 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-30T12:54:45.949 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-03-30T12:54:45.951 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-03-30T12:54:45.952 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-30T12:54:46.096 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-30T12:54:46.107 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: /sbin/restorecon: command not found 2024-03-30T12:54:46.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T12:54:46.108 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-30T12:54:46.164 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-30T12:54:46.164 INFO:tasks.ceph:role: osd.3 2024-03-30T12:54:46.164 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi067.front.sepia.ceph.com 2024-03-30T12:54:46.164 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-30T12:54:46.228 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-30T12:54:46.228 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-30T12:54:46.228 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-30T12:54:46.228 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-30T12:54:46.228 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-30T12:54:46.229 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-03-30T12:54:46.229 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-30T12:54:46.229 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-30T12:54:46.229 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-30T12:54:46.229 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-30T12:54:46.648 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-03-30T12:54:46.651 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-03-30T12:54:46.651 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-30T12:54:46.738 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-30T12:54:46.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T12:54:46.751 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: /sbin/restorecon: command not found 2024-03-30T12:54:46.751 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-30T12:54:46.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:46.858+0000 7ff201b357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:54:46.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:46.858+0000 7ff201b357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:54:46.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:46.862+0000 7ff201b357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:54:46.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:46.862+0000 7ff201b357c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-30T12:54:46.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:46.862+0000 7ff201b357c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-30T12:54:46.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:46.862+0000 7ff201b357c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-30T12:54:46.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:46.862+0000 7ff201b357c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-30T12:54:46.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:46.862+0000 7ff201b357c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-30T12:54:46.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:46.870+0000 7ff201b357c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-30T12:54:49.836 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-30T12:54:49.851 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-30T12:54:49.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:49.930+0000 7fb38ef847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:54:49.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:49.930+0000 7fb38ef847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:54:49.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:49.930+0000 7fb38ef847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:54:49.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:49.934+0000 7fb38ef847c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-30T12:54:49.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:49.934+0000 7fb38ef847c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-30T12:54:49.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:49.934+0000 7fb38ef847c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-30T12:54:49.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:49.934+0000 7fb38ef847c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-30T12:54:49.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:49.934+0000 7fb38ef847c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-30T12:54:49.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:49.934+0000 7fb38ef847c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-30T12:54:52.900 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-30T12:54:52.957 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-30T12:54:53.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:53.038+0000 7ff5a24b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:54:53.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:53.038+0000 7ff5a24b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:54:53.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:53.038+0000 7ff5a24b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:54:53.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:53.038+0000 7ff5a24b77c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-30T12:54:53.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:53.042+0000 7ff5a24b77c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-30T12:54:53.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:53.042+0000 7ff5a24b77c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-30T12:54:53.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:53.042+0000 7ff5a24b77c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-30T12:54:53.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:53.042+0000 7ff5a24b77c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-30T12:54:53.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:53.042+0000 7ff5a24b77c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-30T12:54:56.022 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-30T12:54:56.076 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-30T12:54:56.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:56.158+0000 7f0bb463e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:54:56.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:56.158+0000 7f0bb463e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:54:56.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:56.158+0000 7f0bb463e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:54:56.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:56.158+0000 7f0bb463e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-30T12:54:56.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:56.158+0000 7f0bb463e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-30T12:54:56.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:56.158+0000 7f0bb463e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-30T12:54:56.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:56.158+0000 7f0bb463e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-30T12:54:56.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:56.158+0000 7f0bb463e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-30T12:54:56.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:54:56.162+0000 7f0bb463e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-30T12:54:59.136 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-30T12:54:59.193 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-30T12:54:59.209 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-30T12:54:59.209 INFO:tasks.ceph:role: osd.4 2024-03-30T12:54:59.209 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi082.front.sepia.ceph.com 2024-03-30T12:54:59.209 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-30T12:54:59.272 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-30T12:54:59.273 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-30T12:54:59.273 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-30T12:54:59.273 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-30T12:54:59.273 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-30T12:54:59.273 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2024-03-30T12:54:59.274 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-30T12:54:59.274 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-30T12:54:59.274 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-30T12:54:59.274 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-30T12:54:59.731 INFO:teuthology.orchestra.run.smithi082.stdout:Discarding blocks...Done. 2024-03-30T12:54:59.734 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2024-03-30T12:54:59.734 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-30T12:54:59.826 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-30T12:54:59.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T12:54:59.836 INFO:teuthology.orchestra.run.smithi082.stderr:sudo: /sbin/restorecon: command not found 2024-03-30T12:54:59.837 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-30T12:54:59.890 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-30T12:54:59.891 INFO:tasks.ceph:role: osd.5 2024-03-30T12:54:59.891 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi082.front.sepia.ceph.com 2024-03-30T12:54:59.891 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-30T12:54:59.953 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-30T12:54:59.953 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-30T12:54:59.953 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-30T12:54:59.953 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-30T12:54:59.953 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-30T12:54:59.953 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2024-03-30T12:54:59.953 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-30T12:54:59.953 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-30T12:54:59.953 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-30T12:54:59.953 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-30T12:55:00.418 INFO:teuthology.orchestra.run.smithi082.stdout:Discarding blocks...Done. 2024-03-30T12:55:00.420 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2024-03-30T12:55:00.420 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-30T12:55:00.555 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-30T12:55:00.566 INFO:teuthology.orchestra.run.smithi082.stderr:sudo: /sbin/restorecon: command not found 2024-03-30T12:55:00.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T12:55:00.567 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-30T12:55:00.621 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-30T12:55:00.621 INFO:tasks.ceph:role: osd.6 2024-03-30T12:55:00.621 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi082.front.sepia.ceph.com 2024-03-30T12:55:00.621 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-30T12:55:00.687 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-30T12:55:00.687 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-30T12:55:00.687 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-30T12:55:00.687 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-30T12:55:00.687 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-30T12:55:00.687 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2024-03-30T12:55:00.687 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-30T12:55:00.687 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-30T12:55:00.687 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-30T12:55:00.687 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-30T12:55:01.175 INFO:teuthology.orchestra.run.smithi082.stdout:Discarding blocks...Done. 2024-03-30T12:55:01.177 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2024-03-30T12:55:01.177 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-30T12:55:01.325 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-30T12:55:01.336 INFO:teuthology.orchestra.run.smithi082.stderr:sudo: /sbin/restorecon: command not found 2024-03-30T12:55:01.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T12:55:01.337 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-30T12:55:01.392 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-30T12:55:01.392 INFO:tasks.ceph:role: osd.7 2024-03-30T12:55:01.393 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi082.front.sepia.ceph.com 2024-03-30T12:55:01.393 DEBUG:teuthology.orchestra.run.smithi082:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-30T12:55:01.457 INFO:teuthology.orchestra.run.smithi082.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-30T12:55:01.457 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-30T12:55:01.457 INFO:teuthology.orchestra.run.smithi082.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-30T12:55:01.457 INFO:teuthology.orchestra.run.smithi082.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-30T12:55:01.458 INFO:teuthology.orchestra.run.smithi082.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-30T12:55:01.458 INFO:teuthology.orchestra.run.smithi082.stdout: = sunit=0 swidth=0 blks 2024-03-30T12:55:01.458 INFO:teuthology.orchestra.run.smithi082.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-30T12:55:01.458 INFO:teuthology.orchestra.run.smithi082.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-30T12:55:01.458 INFO:teuthology.orchestra.run.smithi082.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-30T12:55:01.458 INFO:teuthology.orchestra.run.smithi082.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-30T12:55:01.879 INFO:teuthology.orchestra.run.smithi082.stdout:Discarding blocks...Done. 2024-03-30T12:55:01.882 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi082.front.sepia.ceph.com -o noatime 2024-03-30T12:55:01.882 DEBUG:teuthology.orchestra.run.smithi082:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-30T12:55:01.982 DEBUG:teuthology.orchestra.run.smithi082:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-30T12:55:01.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T12:55:01.994 INFO:teuthology.orchestra.run.smithi082.stderr:sudo: /sbin/restorecon: command not found 2024-03-30T12:55:01.994 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-30T12:55:02.097 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:02.096+0000 7f3a6c3ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:02.097 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:02.096+0000 7f3a6c3ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:02.098 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:02.096+0000 7f3a6c3ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:02.099 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:02.096+0000 7f3a6c3ae7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-30T12:55:02.102 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:02.100+0000 7f3a6c3ae7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-30T12:55:02.103 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:02.100+0000 7f3a6c3ae7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-30T12:55:02.103 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:02.100+0000 7f3a6c3ae7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-30T12:55:02.103 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:02.100+0000 7f3a6c3ae7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-30T12:55:02.109 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:02.108+0000 7f3a6c3ae7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-30T12:55:05.084 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-30T12:55:05.098 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-30T12:55:05.179 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:05.175+0000 7fb8694f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:05.180 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:05.175+0000 7fb8694f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:05.181 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:05.179+0000 7fb8694f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:05.182 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:05.179+0000 7fb8694f67c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-30T12:55:05.182 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:05.179+0000 7fb8694f67c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-30T12:55:05.182 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:05.179+0000 7fb8694f67c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-30T12:55:05.182 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:05.179+0000 7fb8694f67c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-30T12:55:05.183 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:05.179+0000 7fb8694f67c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-30T12:55:05.183 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:05.179+0000 7fb8694f67c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-30T12:55:08.156 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-30T12:55:08.208 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-30T12:55:08.287 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:08.283+0000 7efd528577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:08.287 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:08.283+0000 7efd528577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:08.288 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:08.287+0000 7efd528577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:08.289 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:08.287+0000 7efd528577c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-30T12:55:08.290 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:08.287+0000 7efd528577c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-30T12:55:08.290 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:08.287+0000 7efd528577c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-30T12:55:08.290 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:08.287+0000 7efd528577c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-30T12:55:08.290 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:08.287+0000 7efd528577c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-30T12:55:08.291 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:08.287+0000 7efd528577c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-30T12:55:11.277 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-30T12:55:11.332 DEBUG:teuthology.orchestra.run.smithi082:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-30T12:55:11.412 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:11.407+0000 7ff0d1caf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:11.412 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:11.407+0000 7ff0d1caf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:11.413 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:11.411+0000 7ff0d1caf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:11.414 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:11.411+0000 7ff0d1caf7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-30T12:55:11.415 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:11.411+0000 7ff0d1caf7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-30T12:55:11.415 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:11.411+0000 7ff0d1caf7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-30T12:55:11.415 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:11.411+0000 7ff0d1caf7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-30T12:55:11.415 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:11.411+0000 7ff0d1caf7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-30T12:55:11.415 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:11.411+0000 7ff0d1caf7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-30T12:55:14.383 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-30T12:55:14.436 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-30T12:55:14.437 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-30T12:55:14.437 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-30T12:55:14.451 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-30T12:55:14.451 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-30T12:55:14.507 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-30T12:55:14.508 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-30T12:55:14.563 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-30T12:55:14.563 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-30T12:55:14.620 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-30T12:55:14.621 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-30T12:55:14.680 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-30T12:55:14.680 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-30T12:55:14.693 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-30T12:55:14.694 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-30T12:55:14.749 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-30T12:55:14.749 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-30T12:55:14.803 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-30T12:55:14.804 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-30T12:55:14.855 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-30T12:55:14.855 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-30T12:55:14.908 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-30T12:55:14.908 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-30T12:55:14.914 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-30T12:55:14.914 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-30T12:55:14.957 INFO:tasks.ceph:Adding keys to all mons... 2024-03-30T12:55:14.957 DEBUG:teuthology.orchestra.run.smithi067:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-30T12:55:14.959 DEBUG:teuthology.orchestra.run.smithi082:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-30T12:55:15.003 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.y] 2024-03-30T12:55:15.003 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCTCwhmLaiFLxAANzAnp0ud9suaTYIBC9P3Xw== 2024-03-30T12:55:15.003 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.0] 2024-03-30T12:55:15.004 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCWCwhmxpt1MxAARS11izPo8MID/56FV9FmOw== 2024-03-30T12:55:15.004 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.1] 2024-03-30T12:55:15.004 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCZCwhmIw++NxAAo5BdYYX2aIZqosBLjbN6jw== 2024-03-30T12:55:15.004 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.2] 2024-03-30T12:55:15.004 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCdCwhmUHWIAhAAKzAixHuwAMlpYvnterJBCg== 2024-03-30T12:55:15.004 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.3] 2024-03-30T12:55:15.004 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCgCwhmY1ydCRAAelLuzXUgXgHOA2AiVZXNhQ== 2024-03-30T12:55:15.004 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.x] 2024-03-30T12:55:15.004 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCTCwhmMAdKMhAACGA6xH0xuIWtlvKbclsqYQ== 2024-03-30T12:55:15.004 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.4] 2024-03-30T12:55:15.004 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCmCwhmTa0ZBhAAeaQzq+LnBaAM+NUoRjJSAw== 2024-03-30T12:55:15.005 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.5] 2024-03-30T12:55:15.005 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCpCwhmnI8ECxAAloPCTfBLmzP2ThuC8BVQWQ== 2024-03-30T12:55:15.005 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.6] 2024-03-30T12:55:15.005 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCsCwhm28NvERAAeDVmaNR48GQxV2BQJKDeXw== 2024-03-30T12:55:15.005 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.7] 2024-03-30T12:55:15.005 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCvCwhmqd3gGBAAzp76b9Imy1cp8wpQCptUYA== 2024-03-30T12:55:15.005 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-03-30T12:55:15.005 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCTCwhmrdNeNhAAqJkhQaEl+f9RPm6dgdq6yg== 2024-03-30T12:55:15.005 INFO:teuthology.orchestra.run.smithi067.stdout:[client.1] 2024-03-30T12:55:15.005 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCTCwhmla41ORAAcHu+VK73la/zAyaSHnb31g== 2024-03-30T12:55:15.011 INFO:teuthology.orchestra.run.smithi082.stdout:[mgr.y] 2024-03-30T12:55:15.011 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQCTCwhmLaiFLxAANzAnp0ud9suaTYIBC9P3Xw== 2024-03-30T12:55:15.011 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.0] 2024-03-30T12:55:15.011 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQCWCwhmxpt1MxAARS11izPo8MID/56FV9FmOw== 2024-03-30T12:55:15.011 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.1] 2024-03-30T12:55:15.012 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQCZCwhmIw++NxAAo5BdYYX2aIZqosBLjbN6jw== 2024-03-30T12:55:15.012 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.2] 2024-03-30T12:55:15.012 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQCdCwhmUHWIAhAAKzAixHuwAMlpYvnterJBCg== 2024-03-30T12:55:15.013 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.3] 2024-03-30T12:55:15.013 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQCgCwhmY1ydCRAAelLuzXUgXgHOA2AiVZXNhQ== 2024-03-30T12:55:15.013 INFO:teuthology.orchestra.run.smithi082.stdout:[mgr.x] 2024-03-30T12:55:15.013 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQCTCwhmMAdKMhAACGA6xH0xuIWtlvKbclsqYQ== 2024-03-30T12:55:15.013 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.4] 2024-03-30T12:55:15.013 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQCmCwhmTa0ZBhAAeaQzq+LnBaAM+NUoRjJSAw== 2024-03-30T12:55:15.013 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.5] 2024-03-30T12:55:15.013 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQCpCwhmnI8ECxAAloPCTfBLmzP2ThuC8BVQWQ== 2024-03-30T12:55:15.013 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.6] 2024-03-30T12:55:15.013 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQCsCwhm28NvERAAeDVmaNR48GQxV2BQJKDeXw== 2024-03-30T12:55:15.014 INFO:teuthology.orchestra.run.smithi082.stdout:[osd.7] 2024-03-30T12:55:15.014 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQCvCwhmqd3gGBAAzp76b9Imy1cp8wpQCptUYA== 2024-03-30T12:55:15.014 INFO:teuthology.orchestra.run.smithi082.stdout:[client.0] 2024-03-30T12:55:15.014 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQCTCwhmrdNeNhAAqJkhQaEl+f9RPm6dgdq6yg== 2024-03-30T12:55:15.014 INFO:teuthology.orchestra.run.smithi082.stdout:[client.1] 2024-03-30T12:55:15.014 INFO:teuthology.orchestra.run.smithi082.stdout: key = AQCTCwhmla41ORAAcHu+VK73la/zAyaSHnb31g== 2024-03-30T12:55:15.015 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-30T12:55:15.049 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-30T12:55:15.100 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:55:15.103 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:55:15.145 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:55:15.189 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:55:15.232 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:55:15.279 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:55:15.323 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:55:15.365 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:55:15.408 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-30T12:55:15.453 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-30T12:55:15.499 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:55:15.502 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:55:15.580 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:55:15.585 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:55:15.669 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:55:15.671 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:55:15.716 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:55:15.719 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-30T12:55:15.795 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-30T12:55:15.809 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-30T12:55:15.879 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-30T12:55:15.897 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-30T12:55:15.962 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-30T12:55:15.962 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-30T12:55:15.973 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-30T12:55:16.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:16.046+0000 7f19c458dc80 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:16.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:16.046+0000 7f19c458dc80 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:16.060 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-30T12:55:16.114 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-30T12:55:16.166 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-30T12:55:16.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:16.246+0000 7f71f938cc80 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:16.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:16.246+0000 7f71f938cc80 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:16.258 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-30T12:55:16.312 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-30T12:55:16.326 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-30T12:55:16.405 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:16.403+0000 7feffd6a4c80 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:16.406 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:16.403+0000 7feffd6a4c80 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:16.419 DEBUG:teuthology.orchestra.run.smithi082:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-30T12:55:16.471 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-30T12:55:16.474 DEBUG:teuthology.orchestra.run.smithi082:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-30T12:55:16.516 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-30T12:55:16.516 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-30T12:55:16.516 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-30T12:55:16.521 INFO:tasks.ceph.mon.a:Started 2024-03-30T12:55:16.521 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-30T12:55:16.521 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-30T12:55:16.523 INFO:tasks.ceph.mon.c:Started 2024-03-30T12:55:16.524 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-30T12:55:16.524 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-30T12:55:16.562 INFO:tasks.ceph.mon.b:Started 2024-03-30T12:55:16.562 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph config set mgr mgr_pool false --force 2024-03-30T12:55:16.590 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T12:55:16.590+0000 7fbb400fec80 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:16.590 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T12:55:16.590+0000 7fbb400fec80 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:16.590 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T12:55:16.590+0000 7fbb400fec80 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:16.597 INFO:tasks.ceph.mon.c.smithi067.stderr:2024-03-30T12:55:16.598+0000 7fd3167d5c80 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:16.597 INFO:tasks.ceph.mon.c.smithi067.stderr:2024-03-30T12:55:16.598+0000 7fd3167d5c80 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:16.597 INFO:tasks.ceph.mon.c.smithi067.stderr:2024-03-30T12:55:16.598+0000 7fd3167d5c80 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:16.632 INFO:tasks.ceph.mon.b.smithi082.stderr:2024-03-30T12:55:16.627+0000 7f56c80dbc80 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:16.633 INFO:tasks.ceph.mon.b.smithi082.stderr:2024-03-30T12:55:16.627+0000 7f56c80dbc80 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:16.633 INFO:tasks.ceph.mon.b.smithi082.stderr:2024-03-30T12:55:16.631+0000 7f56c80dbc80 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:16.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:16.670+0000 7efcd8e7c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:22.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:22.945+0000 7efcd8e7c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:23.060 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-30T12:55:23.061 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-30T12:55:23.061 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-30T12:55:23.105 INFO:tasks.ceph.mgr.y:Started 2024-03-30T12:55:23.105 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-30T12:55:23.105 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-30T12:55:23.108 INFO:tasks.ceph.mgr.x:Started 2024-03-30T12:55:23.108 INFO:tasks.ceph:Setting crush tunables to default 2024-03-30T12:55:23.108 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd crush tunables default 2024-03-30T12:55:23.202 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-30T12:55:23.201+0000 7f3dac1df180 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:23.209 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-30T12:55:23.209+0000 7f3dac1df180 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:23.209 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-30T12:55:23.209+0000 7f3dac1df180 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:23.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:23.217+0000 7fa6f4e47640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:23.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:23.233+0000 7fa6f4e47640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:23.238 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-30T12:55:23.235+0000 7f373df14180 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:23.250 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-30T12:55:23.247+0000 7f373df14180 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:23.250 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-30T12:55:23.247+0000 7f373df14180 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:23.321 INFO:teuthology.orchestra.run.smithi067.stderr:adjusted tunables profile to default 2024-03-30T12:55:23.340 INFO:tasks.ceph:check_enable_crimson: False 2024-03-30T12:55:23.340 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-30T12:55:23.340 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-30T12:55:23.340 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-30T12:55:23.353 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-30T12:55:23.353 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-30T12:55:23.380 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-30T12:55:23.381+0000 7f3dac1df180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T12:55:23.391 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-30T12:55:23.391 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-30T12:55:23.444 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-30T12:55:23.439+0000 7f373df14180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-30T12:55:23.446 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-03-30T12:55:23.446 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-30T12:55:23.498 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-30T12:55:23.499 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-30T12:55:23.513 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-30T12:55:23.513 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-30T12:55:23.568 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-30T12:55:23.568 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-30T12:55:23.624 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-03-30T12:55:23.624 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-30T12:55:23.679 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 1320388d-ee71-45f3-8a07-cb3f497e9c28 0 2024-03-30T12:55:23.836 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:23.827+0000 7f54410ee640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:23.847 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:23.843+0000 7f54410ee640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:23.893 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-30T12:55:23.893+0000 7f3dac1df180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T12:55:23.918 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-30T12:55:23.915+0000 7f373df14180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-30T12:55:23.939 INFO:teuthology.orchestra.run.smithi082.stdout:0 2024-03-30T12:55:23.956 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 0cd00c37-773f-4a6e-bb54-deed4a0d2a61 1 2024-03-30T12:55:24.071 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:24.059+0000 7f065a7fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:24.080 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:24.075+0000 7f065a7fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:24.088 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-30T12:55:24.089+0000 7f3dac1df180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T12:55:24.103 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-30T12:55:24.099+0000 7f373df14180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-30T12:55:24.162 INFO:teuthology.orchestra.run.smithi082.stdout:1 2024-03-30T12:55:24.184 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 0be6d4c0-5d83-485d-afb7-461810f7efbd 2 2024-03-30T12:55:24.294 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:24.283+0000 7fd3f6d74640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:24.301 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:24.299+0000 7fd3f6d74640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:24.384 INFO:teuthology.orchestra.run.smithi082.stdout:2 2024-03-30T12:55:24.401 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new f851bae3-8485-4e67-8c62-7c9eca784b98 3 2024-03-30T12:55:24.506 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:24.495+0000 7fce44bb1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:24.514 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:24.511+0000 7fce44bb1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:24.590 INFO:teuthology.orchestra.run.smithi082.stdout:3 2024-03-30T12:55:24.608 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 515f79f0-fb30-4438-9668-c32ce26a63fc 4 2024-03-30T12:55:24.660 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-30T12:55:24.661+0000 7f3dac1df180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T12:55:24.673 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-30T12:55:24.671+0000 7f373df14180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-30T12:55:24.713 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:24.703+0000 7f32fabc7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:24.716 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-30T12:55:24.717+0000 7f3dac1df180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T12:55:24.720 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:24.719+0000 7f32fabc7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:24.732 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-30T12:55:24.727+0000 7f373df14180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-30T12:55:24.771 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-30T12:55:24.773+0000 7f3dac1df180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T12:55:24.787 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-30T12:55:24.783+0000 7f373df14180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-30T12:55:24.800 INFO:teuthology.orchestra.run.smithi082.stdout:4 2024-03-30T12:55:24.816 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new e73cd117-12e6-47b5-8561-e5ede18b977c 5 2024-03-30T12:55:24.853 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-30T12:55:24.853+0000 7f3dac1df180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T12:55:24.867 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-30T12:55:24.863+0000 7f373df14180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-30T12:55:24.907 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-30T12:55:24.905+0000 7f3dac1df180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T12:55:24.921 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-30T12:55:24.919+0000 7f373df14180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-30T12:55:24.925 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:24.911+0000 7f53ad607640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:24.933 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:24.931+0000 7f53ad607640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:24.989 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-30T12:55:24.989+0000 7f3dac1df180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T12:55:25.000 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-30T12:55:24.995+0000 7f373df14180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-30T12:55:25.015 INFO:teuthology.orchestra.run.smithi082.stdout:5 2024-03-30T12:55:25.031 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new 9cb2aa10-05de-4b9c-bd43-56773f909acf 6 2024-03-30T12:55:25.045 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-30T12:55:25.045+0000 7f3dac1df180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T12:55:25.056 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-30T12:55:25.051+0000 7f373df14180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-30T12:55:25.134 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:25.123+0000 7fb21c7a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.142 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:25.139+0000 7fb21c7a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.201 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-30T12:55:25.201+0000 7f3dac1df180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T12:55:25.223 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-30T12:55:25.219+0000 7f373df14180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-30T12:55:25.231 INFO:teuthology.orchestra.run.smithi082.stdout:6 2024-03-30T12:55:25.249 DEBUG:teuthology.orchestra.run.smithi082:> sudo ceph --cluster ceph osd new cc3bfce6-25bf-4f70-8f70-ee26ea758ba1 7 2024-03-30T12:55:25.265 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-30T12:55:25.265+0000 7f3dac1df180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T12:55:25.290 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-30T12:55:25.287+0000 7f373df14180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-30T12:55:25.402 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:25.391+0000 7f3dd2a31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.409 INFO:teuthology.orchestra.run.smithi082.stderr:2024-03-30T12:55:25.407+0000 7f3dd2a31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.485 INFO:teuthology.orchestra.run.smithi082.stdout:7 2024-03-30T12:55:25.500 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-30T12:55:25.500 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-30T12:55:25.503 INFO:tasks.ceph.osd.0:Started 2024-03-30T12:55:25.503 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-30T12:55:25.503 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-30T12:55:25.505 INFO:tasks.ceph.osd.1:Started 2024-03-30T12:55:25.505 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-30T12:55:25.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-30T12:55:25.508 INFO:tasks.ceph.osd.2:Started 2024-03-30T12:55:25.508 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-30T12:55:25.508 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-30T12:55:25.510 INFO:tasks.ceph.osd.3:Started 2024-03-30T12:55:25.513 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-30T12:55:25.513 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-30T12:55:25.515 INFO:tasks.ceph.osd.4:Started 2024-03-30T12:55:25.515 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-30T12:55:25.515 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-30T12:55:25.517 INFO:tasks.ceph.osd.5:Started 2024-03-30T12:55:25.517 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-30T12:55:25.517 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-30T12:55:25.518 INFO:tasks.ceph.osd.6:Started 2024-03-30T12:55:25.518 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-30T12:55:25.519 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-30T12:55:25.520 INFO:tasks.ceph.osd.7:Started 2024-03-30T12:55:25.520 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T12:55:25.561 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T12:55:25.561+0000 7fbca12957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.567 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T12:55:25.563+0000 7f75d4f037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.568 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-03-30T12:55:25.569+0000 7f6059ea37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.570 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T12:55:25.567+0000 7f4c3cff67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.571 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T12:55:25.567+0000 7f3b29e3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.574 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T12:55:25.571+0000 7f75d4f037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.574 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T12:55:25.571+0000 7f75d4f037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.579 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T12:55:25.575+0000 7f3b29e3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.579 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T12:55:25.575+0000 7f3b29e3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.579 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T12:55:25.575+0000 7f4c3cff67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.579 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T12:55:25.575+0000 7f4c3cff67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.580 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:55:25.579+0000 7f23c22507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.582 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-03-30T12:55:25.581+0000 7f710e4957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.587 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:55:25.583+0000 7f23c22507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.587 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:55:25.583+0000 7f23c22507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.592 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T12:55:25.593+0000 7fbca12957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.592 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T12:55:25.593+0000 7fbca12957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.593 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-03-30T12:55:25.593+0000 7f6059ea37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.593 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-03-30T12:55:25.593+0000 7f6059ea37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.600 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T12:55:25.597+0000 7f54d47937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.615 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T12:55:25.613+0000 7f54d47937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.615 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T12:55:25.613+0000 7f54d47937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.619 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-03-30T12:55:25.617+0000 7f710e4957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.619 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-03-30T12:55:25.621+0000 7f710e4957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:25.665+0000 7f1ade9b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:25.677+0000 7f1ade9b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:25.744 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:55:25.744 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":10,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T12:55:25.484082+0000","last_up_change":"0.000000","last_in_change":"2024-03-30T12:55:25.484082+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T12:55:25.759 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-30T12:55:25.759 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-30T12:55:25.926 INFO:tasks.ceph.mgr.x.smithi082.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-03-30T12:55:25.927 INFO:tasks.ceph.mgr.x.smithi082.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-03-30T12:55:25.927 INFO:tasks.ceph.mgr.x.smithi082.stderr: from numpy import show_config as show_numpy_config 2024-03-30T12:55:25.936 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T12:55:25.937+0000 7f54d47937c0 -1 Falling back to public interface 2024-03-30T12:55:25.940 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-03-30T12:55:25.941+0000 7f710e4957c0 -1 Falling back to public interface 2024-03-30T12:55:25.967 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-30T12:55:25.963+0000 7f373df14180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T12:55:26.019 INFO:tasks.ceph.mgr.y.smithi067.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-03-30T12:55:26.019 INFO:tasks.ceph.mgr.y.smithi067.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-03-30T12:55:26.019 INFO:tasks.ceph.mgr.y.smithi067.stderr: from numpy import show_config as show_numpy_config 2024-03-30T12:55:26.098 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-30T12:55:26.095+0000 7f373df14180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T12:55:26.117 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-30T12:55:26.117+0000 7f3dac1df180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-30T12:55:26.118 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T12:55:26.115+0000 7f75d4f037c0 -1 Falling back to public interface 2024-03-30T12:55:26.118 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T12:55:26.115+0000 7f4c3cff67c0 -1 Falling back to public interface 2024-03-30T12:55:26.133 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T12:55:26.131+0000 7f3b29e3e7c0 -1 Falling back to public interface 2024-03-30T12:55:26.145 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:55:26.143+0000 7f23c22507c0 -1 Falling back to public interface 2024-03-30T12:55:26.165 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T12:55:26.165+0000 7fbca12957c0 -1 Falling back to public interface 2024-03-30T12:55:26.166 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-03-30T12:55:26.165+0000 7f6059ea37c0 -1 Falling back to public interface 2024-03-30T12:55:26.256 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-30T12:55:26.251+0000 7f373df14180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T12:55:26.310 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-30T12:55:26.309+0000 7f3dac1df180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-30T12:55:26.318 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-30T12:55:26.315+0000 7f373df14180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T12:55:26.451 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-30T12:55:26.447+0000 7f373df14180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T12:55:26.539 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-30T12:55:26.537+0000 7f3dac1df180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-30T12:55:26.562 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-30T12:55:26.611 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-30T12:55:26.609+0000 7f3dac1df180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-30T12:55:26.666 INFO:teuthology.misc.health.smithi067.stderr:2024-03-30T12:55:26.661+0000 7f59eedbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:26.673 INFO:teuthology.misc.health.smithi067.stderr:2024-03-30T12:55:26.673+0000 7f59eedbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:26.754 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-30T12:55:26.754 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T12:55:25.484082+0000","last_up_change":"0.000000","last_in_change":"2024-03-30T12:55:25.484082+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T12:55:26.774 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-30T12:55:26.780 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-30T12:55:26.781+0000 7f3dac1df180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-30T12:55:27.070 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-30T12:55:27.067+0000 7f373df14180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T12:55:27.145 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-30T12:55:27.143+0000 7f373df14180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T12:55:27.323 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-30T12:55:27.319+0000 7f373df14180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T12:55:27.397 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-30T12:55:27.395+0000 7f373df14180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T12:55:27.474 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-30T12:55:27.473+0000 7f3dac1df180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-30T12:55:27.553 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-30T12:55:27.553+0000 7f3dac1df180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-30T12:55:27.682 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-30T12:55:27.679+0000 7f373df14180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T12:55:27.741 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-30T12:55:27.741+0000 7f3dac1df180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-30T12:55:27.743 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-30T12:55:27.739+0000 7f373df14180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T12:55:27.821 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-30T12:55:27.821+0000 7f3dac1df180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-30T12:55:28.130 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-30T12:55:28.129+0000 7f3dac1df180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-30T12:55:28.194 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-03-30T12:55:28.193+0000 7f3dac1df180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-30T12:55:28.795 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-30T12:55:28.791+0000 7f36e5461640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-30T12:55:28.795 INFO:tasks.ceph.mgr.x.smithi082.stderr:2024-03-30T12:55:28.791+0000 7f36e5461640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-30T12:55:29.760 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T12:55:29.761+0000 7f54d47937c0 -1 osd.3 0 log_to_monitors true 2024-03-30T12:55:29.870 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-03-30T12:55:29.869+0000 7f710e4957c0 -1 osd.0 0 log_to_monitors true 2024-03-30T12:55:29.958 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T12:55:29.955+0000 7f4c3cff67c0 -1 osd.4 0 log_to_monitors true 2024-03-30T12:55:29.971 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-03-30T12:55:29.969+0000 7f6059ea37c0 -1 osd.2 0 log_to_monitors true 2024-03-30T12:55:29.988 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:55:29.983+0000 7f23c22507c0 -1 osd.7 0 log_to_monitors true 2024-03-30T12:55:30.079 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T12:55:30.081+0000 7fbca12957c0 -1 osd.1 0 log_to_monitors true 2024-03-30T12:55:30.085 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T12:55:30.083+0000 7f3b29e3e7c0 -1 osd.5 0 log_to_monitors true 2024-03-30T12:55:30.086 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T12:55:30.083+0000 7f75d4f037c0 -1 osd.6 0 log_to_monitors true 2024-03-30T12:55:30.808 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T12:55:30.809+0000 7f54cff19640 -1 osd.3 0 waiting for initial osdmap 2024-03-30T12:55:30.821 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T12:55:30.821+0000 7f54cc568640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T12:55:31.803 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:55:31.799+0000 7f23bd9d6640 -1 osd.7 0 waiting for initial osdmap 2024-03-30T12:55:31.804 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T12:55:31.803+0000 7f3b255c4640 -1 osd.5 0 waiting for initial osdmap 2024-03-30T12:55:31.836 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:55:31.835+0000 7f23ba025640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T12:55:31.849 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T12:55:31.847+0000 7f3b21c13640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T12:55:33.026 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T12:55:33.019+0000 7f75d0689640 -1 osd.6 0 waiting for initial osdmap 2024-03-30T12:55:33.041 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T12:55:33.039+0000 7f75c8cd0640 -1 osd.6 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T12:55:33.046 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T12:55:33.043+0000 7f4c39fb2640 -1 osd.4 0 waiting for initial osdmap 2024-03-30T12:55:33.069 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T12:55:33.067+0000 7f4c30dc3640 -1 osd.4 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T12:55:33.418 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T12:55:33.417+0000 7fbc9e251640 -1 osd.1 0 waiting for initial osdmap 2024-03-30T12:55:33.449 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T12:55:33.449+0000 7fbc95062640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T12:55:33.472 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-03-30T12:55:33.473+0000 7f6055629640 -1 osd.2 0 waiting for initial osdmap 2024-03-30T12:55:33.473 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-03-30T12:55:33.473+0000 7f7109c1b640 -1 osd.0 0 waiting for initial osdmap 2024-03-30T12:55:33.497 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-03-30T12:55:33.497+0000 7f7102262640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T12:55:33.498 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-03-30T12:55:33.497+0000 7f604dc70640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T12:55:33.581 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-30T12:55:33.703 INFO:teuthology.misc.health.smithi067.stderr:2024-03-30T12:55:33.697+0000 7f1849ea6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:33.710 INFO:teuthology.misc.health.smithi067.stderr:2024-03-30T12:55:33.709+0000 7f1849ea6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:33.897 INFO:teuthology.misc.health.smithi067.stdout: 2024-03-30T12:55:33.897 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":15,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T12:55:33.836470+0000","last_up_change":"2024-03-30T12:55:33.836470+0000","last_in_change":"2024-03-30T12:55:25.484082+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3146012944},{"type":"v1","addr":"172.21.15.67:6818","nonce":3146012944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3146012944},{"type":"v1","addr":"172.21.15.67:6821","nonce":3146012944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3146012944},{"type":"v1","addr":"172.21.15.67:6828","nonce":3146012944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":3146012944},{"type":"v1","addr":"172.21.15.67:6825","nonce":3146012944}]},"public_addr":"172.21.15.67:6818/3146012944","cluster_addr":"172.21.15.67:6821/3146012944","heartbeat_back_addr":"172.21.15.67:6828/3146012944","heartbeat_front_addr":"172.21.15.67:6825/3146012944","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6801","nonce":1504348724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6803","nonce":1504348724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6807","nonce":1504348724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6805","nonce":1504348724}]},"public_addr":"172.21.15.67:6801/1504348724","cluster_addr":"172.21.15.67:6803/1504348724","heartbeat_back_addr":"172.21.15.67:6807/1504348724","heartbeat_front_addr":"172.21.15.67:6805/1504348724","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1760616118},{"type":"v1","addr":"172.21.15.82:6803","nonce":1760616118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":1760616118},{"type":"v1","addr":"172.21.15.82:6807","nonce":1760616118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":1760616118},{"type":"v1","addr":"172.21.15.82:6815","nonce":1760616118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":1760616118},{"type":"v1","addr":"172.21.15.82:6811","nonce":1760616118}]},"public_addr":"172.21.15.82:6803/1760616118","cluster_addr":"172.21.15.82:6807/1760616118","heartbeat_back_addr":"172.21.15.82:6815/1760616118","heartbeat_front_addr":"172.21.15.82:6811/1760616118","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6817","nonce":3467856022}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6819","nonce":3467856022}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6823","nonce":3467856022}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6821","nonce":3467856022}]},"public_addr":"172.21.15.82:6817/3467856022","cluster_addr":"172.21.15.82:6819/3467856022","heartbeat_back_addr":"172.21.15.82:6823/3467856022","heartbeat_front_addr":"172.21.15.82:6821/3467856022","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6801","nonce":968508548}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6806","nonce":968508548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6814","nonce":968508548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6810","nonce":968508548}]},"public_addr":"172.21.15.82:6801/968508548","cluster_addr":"172.21.15.82:6806/968508548","heartbeat_back_addr":"172.21.15.82:6814/968508548","heartbeat_front_addr":"172.21.15.82:6810/968508548","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2471694698},{"type":"v1","addr":"172.21.15.82:6825","nonce":2471694698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2471694698},{"type":"v1","addr":"172.21.15.82:6827","nonce":2471694698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2471694698},{"type":"v1","addr":"172.21.15.82:6831","nonce":2471694698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2471694698},{"type":"v1","addr":"172.21.15.82:6829","nonce":2471694698}]},"public_addr":"172.21.15.82:6825/2471694698","cluster_addr":"172.21.15.82:6827/2471694698","heartbeat_back_addr":"172.21.15.82:6831/2471694698","heartbeat_front_addr":"172.21.15.82:6829/2471694698","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:55:30.724117+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:55:31.041350+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:55:31.018547+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T12:55:33.915 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-30T12:55:33.915 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-30T12:55:33.915 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-30T12:55:33.916 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-30T12:55:33.916 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-30T12:55:33.916 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-30T12:55:34.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:34.033+0000 7f68a9299640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:34.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:34.045+0000 7f68a9299640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:34.273 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:55:34.293 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":4,"flags":0,"active_gid":4114,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":953849964},{"type":"v1","addr":"172.21.15.82:6833","nonce":953849964}]},"active_addr":"172.21.15.82:6833/953849964","active_change":"2024-03-30T12:55:27.769772+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4106,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.82:0","nonce":1444242724}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.82:0","nonce":355057847}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.82:0","nonce":3952669618}]}]} 2024-03-30T12:55:34.297 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-30T12:55:34.297 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-30T12:55:34.297 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T12:55:34.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:34.401+0000 7f4ab7cf2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:34.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:34.413+0000 7f4ab7cf2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:34.592 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:55:34.593 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":15,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T12:55:33.836470+0000","last_up_change":"2024-03-30T12:55:33.836470+0000","last_in_change":"2024-03-30T12:55:25.484082+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3146012944},{"type":"v1","addr":"172.21.15.67:6818","nonce":3146012944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3146012944},{"type":"v1","addr":"172.21.15.67:6821","nonce":3146012944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3146012944},{"type":"v1","addr":"172.21.15.67:6828","nonce":3146012944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":3146012944},{"type":"v1","addr":"172.21.15.67:6825","nonce":3146012944}]},"public_addr":"172.21.15.67:6818/3146012944","cluster_addr":"172.21.15.67:6821/3146012944","heartbeat_back_addr":"172.21.15.67:6828/3146012944","heartbeat_front_addr":"172.21.15.67:6825/3146012944","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6801","nonce":1504348724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6803","nonce":1504348724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6807","nonce":1504348724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6805","nonce":1504348724}]},"public_addr":"172.21.15.67:6801/1504348724","cluster_addr":"172.21.15.67:6803/1504348724","heartbeat_back_addr":"172.21.15.67:6807/1504348724","heartbeat_front_addr":"172.21.15.67:6805/1504348724","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1760616118},{"type":"v1","addr":"172.21.15.82:6803","nonce":1760616118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":1760616118},{"type":"v1","addr":"172.21.15.82:6807","nonce":1760616118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":1760616118},{"type":"v1","addr":"172.21.15.82:6815","nonce":1760616118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":1760616118},{"type":"v1","addr":"172.21.15.82:6811","nonce":1760616118}]},"public_addr":"172.21.15.82:6803/1760616118","cluster_addr":"172.21.15.82:6807/1760616118","heartbeat_back_addr":"172.21.15.82:6815/1760616118","heartbeat_front_addr":"172.21.15.82:6811/1760616118","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6817","nonce":3467856022}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6819","nonce":3467856022}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6823","nonce":3467856022}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6821","nonce":3467856022}]},"public_addr":"172.21.15.82:6817/3467856022","cluster_addr":"172.21.15.82:6819/3467856022","heartbeat_back_addr":"172.21.15.82:6823/3467856022","heartbeat_front_addr":"172.21.15.82:6821/3467856022","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6801","nonce":968508548}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6806","nonce":968508548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6814","nonce":968508548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6810","nonce":968508548}]},"public_addr":"172.21.15.82:6801/968508548","cluster_addr":"172.21.15.82:6806/968508548","heartbeat_back_addr":"172.21.15.82:6814/968508548","heartbeat_front_addr":"172.21.15.82:6810/968508548","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2471694698},{"type":"v1","addr":"172.21.15.82:6825","nonce":2471694698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2471694698},{"type":"v1","addr":"172.21.15.82:6827","nonce":2471694698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2471694698},{"type":"v1","addr":"172.21.15.82:6831","nonce":2471694698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2471694698},{"type":"v1","addr":"172.21.15.82:6829","nonce":2471694698}]},"public_addr":"172.21.15.82:6825/2471694698","cluster_addr":"172.21.15.82:6827/2471694698","heartbeat_back_addr":"172.21.15.82:6831/2471694698","heartbeat_front_addr":"172.21.15.82:6829/2471694698","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:55:30.724117+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:55:31.041350+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:55:31.018547+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T12:55:34.610 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-30T12:55:34.611 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T12:55:34.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:34.729+0000 7f09656af640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:34.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:34.741+0000 7f09656af640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:34.912 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:55:34.912 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":16,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T12:55:34.847810+0000","last_up_change":"2024-03-30T12:55:33.836470+0000","last_in_change":"2024-03-30T12:55:25.484082+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3146012944},{"type":"v1","addr":"172.21.15.67:6818","nonce":3146012944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3146012944},{"type":"v1","addr":"172.21.15.67:6821","nonce":3146012944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3146012944},{"type":"v1","addr":"172.21.15.67:6828","nonce":3146012944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":3146012944},{"type":"v1","addr":"172.21.15.67:6825","nonce":3146012944}]},"public_addr":"172.21.15.67:6818/3146012944","cluster_addr":"172.21.15.67:6821/3146012944","heartbeat_back_addr":"172.21.15.67:6828/3146012944","heartbeat_front_addr":"172.21.15.67:6825/3146012944","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6801","nonce":1504348724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6803","nonce":1504348724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6807","nonce":1504348724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6805","nonce":1504348724}]},"public_addr":"172.21.15.67:6801/1504348724","cluster_addr":"172.21.15.67:6803/1504348724","heartbeat_back_addr":"172.21.15.67:6807/1504348724","heartbeat_front_addr":"172.21.15.67:6805/1504348724","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1760616118},{"type":"v1","addr":"172.21.15.82:6803","nonce":1760616118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":1760616118},{"type":"v1","addr":"172.21.15.82:6807","nonce":1760616118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":1760616118},{"type":"v1","addr":"172.21.15.82:6815","nonce":1760616118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":1760616118},{"type":"v1","addr":"172.21.15.82:6811","nonce":1760616118}]},"public_addr":"172.21.15.82:6803/1760616118","cluster_addr":"172.21.15.82:6807/1760616118","heartbeat_back_addr":"172.21.15.82:6815/1760616118","heartbeat_front_addr":"172.21.15.82:6811/1760616118","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6817","nonce":3467856022}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6819","nonce":3467856022}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6823","nonce":3467856022}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6821","nonce":3467856022}]},"public_addr":"172.21.15.82:6817/3467856022","cluster_addr":"172.21.15.82:6819/3467856022","heartbeat_back_addr":"172.21.15.82:6823/3467856022","heartbeat_front_addr":"172.21.15.82:6821/3467856022","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6801","nonce":968508548}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6806","nonce":968508548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6814","nonce":968508548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6810","nonce":968508548}]},"public_addr":"172.21.15.82:6801/968508548","cluster_addr":"172.21.15.82:6806/968508548","heartbeat_back_addr":"172.21.15.82:6814/968508548","heartbeat_front_addr":"172.21.15.82:6810/968508548","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2471694698},{"type":"v1","addr":"172.21.15.82:6825","nonce":2471694698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2471694698},{"type":"v1","addr":"172.21.15.82:6827","nonce":2471694698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2471694698},{"type":"v1","addr":"172.21.15.82:6831","nonce":2471694698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2471694698},{"type":"v1","addr":"172.21.15.82:6829","nonce":2471694698}]},"public_addr":"172.21.15.82:6825/2471694698","cluster_addr":"172.21.15.82:6827/2471694698","heartbeat_back_addr":"172.21.15.82:6831/2471694698","heartbeat_front_addr":"172.21.15.82:6829/2471694698","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:55:30.919494+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:55:31.132465+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:55:30.980874+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:55:30.724117+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:55:30.946527+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:55:31.041350+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:55:31.124729+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:55:31.018547+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T12:55:34.931 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-30T12:55:34.932 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-30T12:55:34.932 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-30T12:55:34.932 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-30T12:55:34.933 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-30T12:55:34.933 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-30T12:55:34.933 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-30T12:55:34.934 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-30T12:55:35.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.101+0000 7f674420b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.105+0000 7f6571240640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.105+0000 7f771ee93640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.105+0000 7f94e1888640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.101+0000 7fe1a5873640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.105+0000 7f739224e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.105+0000 7f7f25ac8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.105+0000 7f34feaf9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.117+0000 7f6742fa9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.121+0000 7f6571240640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.125+0000 7f739224e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.125+0000 7f34feaf9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.125+0000 7fe19ffff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.125+0000 7f94e1888640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.125+0000 7f771dc31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.125+0000 7f7f25ac8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.197 INFO:teuthology.orchestra.run.smithi067.stdout:60129542147 2024-03-30T12:55:35.197 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-30T12:55:35.207 INFO:teuthology.orchestra.run.smithi067.stdout:64424509443 2024-03-30T12:55:35.207 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-30T12:55:35.210 INFO:teuthology.orchestra.run.smithi067.stdout:64424509443 2024-03-30T12:55:35.210 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-30T12:55:35.211 INFO:teuthology.orchestra.run.smithi067.stdout:60129542147 2024-03-30T12:55:35.212 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-30T12:55:35.213 INFO:teuthology.orchestra.run.smithi067.stdout:64424509444 2024-03-30T12:55:35.213 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-30T12:55:35.214 INFO:teuthology.orchestra.run.smithi067.stdout:64424509444 2024-03-30T12:55:35.215 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-30T12:55:35.216 INFO:teuthology.orchestra.run.smithi067.stdout:55834574852 2024-03-30T12:55:35.216 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-30T12:55:35.217 INFO:teuthology.orchestra.run.smithi067.stdout:64424509444 2024-03-30T12:55:35.217 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-30T12:55:35.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.337+0000 7f4214b7c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.345+0000 7f116680d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.345+0000 7f81fada8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.349+0000 7f5ba994c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.357+0000 7f4214b7c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.369+0000 7f5ba994c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.369+0000 7f116680d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.381+0000 7f81f9b46640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.413+0000 7f24f6c02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.417+0000 7fac53f6b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.417+0000 7f400951b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.421+0000 7f0e7da40640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.433+0000 7f24f59a0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.437+0000 7fac53f6b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.441+0000 7f400951b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:35.445+0000 7f0e7da40640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:35.706 INFO:teuthology.orchestra.run.smithi067.stdout:0 2024-03-30T12:55:35.711 INFO:teuthology.orchestra.run.smithi067.stdout:0 2024-03-30T12:55:35.718 INFO:teuthology.orchestra.run.smithi067.stdout:0 2024-03-30T12:55:35.718 INFO:teuthology.orchestra.run.smithi067.stdout:0 2024-03-30T12:55:35.738 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542147 got 0 for osd.7 2024-03-30T12:55:35.746 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 0 for osd.0 2024-03-30T12:55:35.747 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 0 for osd.6 2024-03-30T12:55:35.758 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542147 got 0 for osd.5 2024-03-30T12:55:35.764 INFO:teuthology.orchestra.run.smithi067.stdout:0 2024-03-30T12:55:35.771 INFO:teuthology.orchestra.run.smithi067.stdout:55834574851 2024-03-30T12:55:35.784 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 0 for osd.2 2024-03-30T12:55:35.785 INFO:teuthology.orchestra.run.smithi067.stdout:0 2024-03-30T12:55:35.789 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.3 2024-03-30T12:55:35.802 INFO:teuthology.orchestra.run.smithi067.stdout:0 2024-03-30T12:55:35.804 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 0 for osd.4 2024-03-30T12:55:35.819 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 0 for osd.1 2024-03-30T12:55:36.739 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-30T12:55:36.747 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-30T12:55:36.748 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-30T12:55:36.759 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-30T12:55:36.785 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-30T12:55:36.790 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-30T12:55:36.805 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-30T12:55:36.820 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-30T12:55:36.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:36.893+0000 7fbc01851640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:36.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:36.909+0000 7fbc01851640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:36.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:36.941+0000 7f7f432ee640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:36.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:36.945+0000 7f5b17d4e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:36.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:36.961+0000 7f7f4208c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:36.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:36.965+0000 7f5b17d4e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:36.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:36.957+0000 7f5028b46640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:36.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:36.985+0000 7f5028b46640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:36.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:36.993+0000 7ff1e5d18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:37.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:36.989+0000 7faf42123640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:37.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:36.997+0000 7f467a9e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:37.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:37.017+0000 7ff1e5d18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:37.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:37.021+0000 7faf42123640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:37.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:37.025+0000 7f467a9e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:37.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:37.029+0000 7f55e6887640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:37.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:37.049+0000 7f55e5625640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:37.241 INFO:teuthology.orchestra.run.smithi067.stdout:60129542147 2024-03-30T12:55:37.263 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542147 got 60129542147 for osd.7 2024-03-30T12:55:37.264 DEBUG:teuthology.parallel:result is None 2024-03-30T12:55:37.295 INFO:teuthology.orchestra.run.smithi067.stdout:60129542147 2024-03-30T12:55:37.303 INFO:teuthology.orchestra.run.smithi067.stdout:64424509443 2024-03-30T12:55:37.311 INFO:teuthology.orchestra.run.smithi067.stdout:64424509443 2024-03-30T12:55:37.325 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542147 got 60129542147 for osd.5 2024-03-30T12:55:37.325 DEBUG:teuthology.parallel:result is None 2024-03-30T12:55:37.326 INFO:teuthology.orchestra.run.smithi067.stdout:55834574852 2024-03-30T12:55:37.333 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509443 for osd.0 2024-03-30T12:55:37.333 DEBUG:teuthology.parallel:result is None 2024-03-30T12:55:37.340 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509443 got 64424509443 for osd.6 2024-03-30T12:55:37.340 DEBUG:teuthology.parallel:result is None 2024-03-30T12:55:37.345 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.3 2024-03-30T12:55:37.345 DEBUG:teuthology.parallel:result is None 2024-03-30T12:55:37.349 INFO:teuthology.orchestra.run.smithi067.stdout:64424509444 2024-03-30T12:55:37.365 INFO:teuthology.orchestra.run.smithi067.stdout:64424509444 2024-03-30T12:55:37.366 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509444 for osd.2 2024-03-30T12:55:37.367 INFO:teuthology.orchestra.run.smithi067.stdout:64424509444 2024-03-30T12:55:37.367 DEBUG:teuthology.parallel:result is None 2024-03-30T12:55:37.383 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509444 for osd.1 2024-03-30T12:55:37.383 DEBUG:teuthology.parallel:result is None 2024-03-30T12:55:37.385 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509444 for osd.4 2024-03-30T12:55:37.385 DEBUG:teuthology.parallel:result is None 2024-03-30T12:55:37.385 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-30T12:55:37.385 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:55:37.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:37.537+0000 7fadad748640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:37.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:37.549+0000 7fadad748640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:37.710 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:55:37.710 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T12:55:37.726 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-03-30T12:55:35.780694+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":0,"kb":754974720,"kb_used":217252,"kb_used_data":1080,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757468,"statfs":{"total":773094113280,"available":772871647232,"internally_reserved":0,"allocated":1105920,"data_stored":291354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509443,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27152,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":135168,"data_stored":35404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542147,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27164,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":147456,"data_stored":40325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27168,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":147456,"data_stored":40325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27152,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":135168,"data_stored":35404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27168,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":147456,"data_stored":40325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509443,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27152,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":135168,"data_stored":35404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542147,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":28763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27156,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":135168,"data_stored":35404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[]}} 2024-03-30T12:55:37.727 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:55:37.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:37.841+0000 7f9f38bbc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:37.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:37.853+0000 7f9f38bbc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:38.019 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:55:38.019 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T12:55:38.037 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-03-30T12:55:37.781207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":0,"kb":754974720,"kb_used":217252,"kb_used_data":1080,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757468,"statfs":{"total":773094113280,"available":772871647232,"internally_reserved":0,"allocated":1105920,"data_stored":291354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509443,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27152,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":135168,"data_stored":35404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542147,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27164,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":147456,"data_stored":40325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27168,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":147456,"data_stored":40325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27152,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":135168,"data_stored":35404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27168,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":147456,"data_stored":40325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509443,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27152,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":135168,"data_stored":35404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542147,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":28763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27156,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":135168,"data_stored":35404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[]}} 2024-03-30T12:55:38.037 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-30T12:55:38.037 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-30T12:55:38.037 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-30T12:55:38.038 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-30T12:55:38.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:38.141+0000 7fd07ba7f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:38.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:38.157+0000 7fd07ba7f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:38.354 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:55:38.354 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-30T12:55:38.372 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-30T12:55:38.373 INFO:teuthology.run_tasks:Running task thrashosds... 2024-03-30T12:55:38.384 INFO:tasks.thrashosds:config is {'chance_test_min_size': 3, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-30T12:55:38.384 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-ceph_test_rados-partial-reads'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': '*', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 2, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'create_rbd_pool': False, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'sha1': '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc'}}, 'rados': {'wait_for_all_active_clean_pgs': True}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5, 'disable_objectstore_tool_tests': True}, 'workunit': {'branch': 'wip-ceph_test_rados-partial-reads', 'sha1': '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc'}} 2024-03-30T12:55:38.384 INFO:tasks.thrashosds:config is {'chance_test_min_size': 3, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5, 'disable_objectstore_tool_tests': True} 2024-03-30T12:55:38.385 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-03-30T12:55:38.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:38.489+0000 7f60d75fb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:38.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:38.505+0000 7f60d75fb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:38.701 INFO:tasks.thrashosds:Beginning thrashosds... 2024-03-30T12:55:38.701 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-30T12:55:38.702 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:55:38.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:38.805+0000 7f6eb4784640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:38.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:38.817+0000 7f6eb4784640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:38.980 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:55:38.981 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T12:55:38.998 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-03-30T12:55:37.781207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":0,"kb":754974720,"kb_used":217252,"kb_used_data":1080,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757468,"statfs":{"total":773094113280,"available":772871647232,"internally_reserved":0,"allocated":1105920,"data_stored":291354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509443,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27152,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":135168,"data_stored":35404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542147,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27164,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":147456,"data_stored":40325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27168,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":147456,"data_stored":40325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27152,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":135168,"data_stored":35404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27168,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":147456,"data_stored":40325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509443,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27152,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":135168,"data_stored":35404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542147,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":28763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27156,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":135168,"data_stored":35404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[]}} 2024-03-30T12:55:38.998 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:55:39.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:39.101+0000 7f3365450640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:39.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:39.113+0000 7f3365450640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:39.282 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:55:39.282 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T12:55:39.300 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-03-30T12:55:37.781207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":0,"kb":754974720,"kb_used":217252,"kb_used_data":1080,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757468,"statfs":{"total":773094113280,"available":772871647232,"internally_reserved":0,"allocated":1105920,"data_stored":291354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[],"pool_stats":[],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509443,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27152,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":135168,"data_stored":35404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542147,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27164,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":147456,"data_stored":40325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27168,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":147456,"data_stored":40325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27152,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":135168,"data_stored":35404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509444,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27168,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":147456,"data_stored":40325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509443,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27152,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":135168,"data_stored":35404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542147,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":28763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27156,"kb_used_data":132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":135168,"data_stored":35404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]}],"pool_statfs":[]}} 2024-03-30T12:55:39.300 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-30T12:55:39.301 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-03-30T12:55:39.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:39.405+0000 7f4bd4f59640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:39.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:39.417+0000 7f4bd4f59640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:39.584 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 16 2024-03-30T12:55:39.585 INFO:teuthology.orchestra.run.smithi067.stdout:fsid 0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a 2024-03-30T12:55:39.585 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-03-30T12:55:22.904201+0000 2024-03-30T12:55:39.585 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-03-30T12:55:34.847810+0000 2024-03-30T12:55:39.585 INFO:teuthology.orchestra.run.smithi067.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-30T12:55:39.585 INFO:teuthology.orchestra.run.smithi067.stdout:crush_version 5 2024-03-30T12:55:39.585 INFO:teuthology.orchestra.run.smithi067.stdout:full_ratio 0.95 2024-03-30T12:55:39.585 INFO:teuthology.orchestra.run.smithi067.stdout:backfillfull_ratio 0.9 2024-03-30T12:55:39.585 INFO:teuthology.orchestra.run.smithi067.stdout:nearfull_ratio 0.85 2024-03-30T12:55:39.585 INFO:teuthology.orchestra.run.smithi067.stdout:require_min_compat_client luminous 2024-03-30T12:55:39.585 INFO:teuthology.orchestra.run.smithi067.stdout:min_compat_client jewel 2024-03-30T12:55:39.585 INFO:teuthology.orchestra.run.smithi067.stdout:require_osd_release squid 2024-03-30T12:55:39.586 INFO:teuthology.orchestra.run.smithi067.stdout:stretch_mode_enabled false 2024-03-30T12:55:39.586 INFO:teuthology.orchestra.run.smithi067.stdout:max_osd 8 2024-03-30T12:55:39.586 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6808/2813590964,v1:172.21.15.67:6809/2813590964] [v2:172.21.15.67:6810/2813590964,v1:172.21.15.67:6811/2813590964] exists,up 1320388d-ee71-45f3-8a07-cb3f497e9c28 2024-03-30T12:55:39.586 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6816/3146012944,v1:172.21.15.67:6818/3146012944] [v2:172.21.15.67:6820/3146012944,v1:172.21.15.67:6821/3146012944] exists,up 0cd00c37-773f-4a6e-bb54-deed4a0d2a61 2024-03-30T12:55:39.586 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6817/3696364776,v1:172.21.15.67:6819/3696364776] [v2:172.21.15.67:6822/3696364776,v1:172.21.15.67:6824/3696364776] exists,up 0be6d4c0-5d83-485d-afb7-461810f7efbd 2024-03-30T12:55:39.586 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6800/1504348724,v1:172.21.15.67:6801/1504348724] [v2:172.21.15.67:6802/1504348724,v1:172.21.15.67:6803/1504348724] exists,up f851bae3-8485-4e67-8c62-7c9eca784b98 2024-03-30T12:55:39.586 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6802/1760616118,v1:172.21.15.82:6803/1760616118] [v2:172.21.15.82:6805/1760616118,v1:172.21.15.82:6807/1760616118] exists,up 515f79f0-fb30-4438-9668-c32ce26a63fc 2024-03-30T12:55:39.586 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6816/3467856022,v1:172.21.15.82:6817/3467856022] [v2:172.21.15.82:6818/3467856022,v1:172.21.15.82:6819/3467856022] exists,up e73cd117-12e6-47b5-8561-e5ede18b977c 2024-03-30T12:55:39.586 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6800/968508548,v1:172.21.15.82:6801/968508548] [v2:172.21.15.82:6804/968508548,v1:172.21.15.82:6806/968508548] exists,up 9cb2aa10-05de-4b9c-bd43-56773f909acf 2024-03-30T12:55:39.587 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6824/2471694698,v1:172.21.15.82:6825/2471694698] [v2:172.21.15.82:6826/2471694698,v1:172.21.15.82:6827/2471694698] exists,up cc3bfce6-25bf-4f70-8f70-ee26ea758ba1 2024-03-30T12:55:39.603 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6808/2813590964,v1:172.21.15.67:6809/2813590964] [v2:172.21.15.67:6810/2813590964,v1:172.21.15.67:6811/2813590964] exists,up 1320388d-ee71-45f3-8a07-cb3f497e9c28', 'osd.1 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6816/3146012944,v1:172.21.15.67:6818/3146012944] [v2:172.21.15.67:6820/3146012944,v1:172.21.15.67:6821/3146012944] exists,up 0cd00c37-773f-4a6e-bb54-deed4a0d2a61', 'osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6817/3696364776,v1:172.21.15.67:6819/3696364776] [v2:172.21.15.67:6822/3696364776,v1:172.21.15.67:6824/3696364776] exists,up 0be6d4c0-5d83-485d-afb7-461810f7efbd', 'osd.3 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6800/1504348724,v1:172.21.15.67:6801/1504348724] [v2:172.21.15.67:6802/1504348724,v1:172.21.15.67:6803/1504348724] exists,up f851bae3-8485-4e67-8c62-7c9eca784b98', 'osd.4 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6802/1760616118,v1:172.21.15.82:6803/1760616118] [v2:172.21.15.82:6805/1760616118,v1:172.21.15.82:6807/1760616118] exists,up 515f79f0-fb30-4438-9668-c32ce26a63fc', 'osd.5 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6816/3467856022,v1:172.21.15.82:6817/3467856022] [v2:172.21.15.82:6818/3467856022,v1:172.21.15.82:6819/3467856022] exists,up e73cd117-12e6-47b5-8561-e5ede18b977c', 'osd.6 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6800/968508548,v1:172.21.15.82:6801/968508548] [v2:172.21.15.82:6804/968508548,v1:172.21.15.82:6806/968508548] exists,up 9cb2aa10-05de-4b9c-bd43-56773f909acf', 'osd.7 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6824/2471694698,v1:172.21.15.82:6825/2471694698] [v2:172.21.15.82:6826/2471694698,v1:172.21.15.82:6827/2471694698] exists,up cc3bfce6-25bf-4f70-8f70-ee26ea758ba1'] 2024-03-30T12:55:39.604 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-03-30T12:55:39.720 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-30T12:55:39.720 INFO:teuthology.orchestra.run.smithi067.stdout: "name": "mon.a", 2024-03-30T12:55:39.720 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster": "ceph", 2024-03-30T12:55:39.720 INFO:teuthology.orchestra.run.smithi067.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-03-30T12:55:39.720 INFO:teuthology.orchestra.run.smithi067.stdout: "admin_socket_mode": "", 2024-03-30T12:55:39.720 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-03-30T12:55:39.720 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_client_required": "cephx, none", 2024-03-30T12:55:39.720 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_cluster_required": "cephx", 2024-03-30T12:55:39.720 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_debug": "true", 2024-03-30T12:55:39.720 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-03-30T12:55:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-03-30T12:55:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_service_required": "cephx", 2024-03-30T12:55:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_service_ticket_ttl": "240.000000", 2024-03-30T12:55:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_supported": "cephx", 2024-03-30T12:55:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_aio": "true", 2024-03-30T12:55:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_aio_max_queue_depth": "1024", 2024-03-30T12:55:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_aio_poll_ms": "250", 2024-03-30T12:55:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_aio_reap_max": "16", 2024-03-30T12:55:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_async_discard": "false", 2024-03-30T12:55:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_block_size": "4096", 2024-03-30T12:55:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_debug_aio": "false", 2024-03-30T12:55:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-03-30T12:55:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-03-30T12:55:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_debug_inflight_ios": "false", 2024-03-30T12:55:39.721 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_enable_discard": "false", 2024-03-30T12:55:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_flock_retry": "3", 2024-03-30T12:55:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_flock_retry_interval": "0.100000", 2024-03-30T12:55:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_inject_crash": "0", 2024-03-30T12:55:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_inject_crash_flush_delay": "2", 2024-03-30T12:55:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_ioring": "false", 2024-03-30T12:55:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_ioring_hipri": "false", 2024-03-30T12:55:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_ioring_sqthread_poll": "false", 2024-03-30T12:55:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-03-30T12:55:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_read_buffer_alignment": "4096", 2024-03-30T12:55:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-03-30T12:55:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: "bdev_type": "", 2024-03-30T12:55:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_alloc_size": "1048576", 2024-03-30T12:55:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_allocator": "hybrid", 2024-03-30T12:55:39.722 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_buffered_io": "true", 2024-03-30T12:55:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_check_for_zeros": "false", 2024-03-30T12:55:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_check_volume_selector_often": "false", 2024-03-30T12:55:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-03-30T12:55:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_compact_log_sync": "false", 2024-03-30T12:55:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-03-30T12:55:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-03-30T12:55:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_log_compact_min_size": "16777216", 2024-03-30T12:55:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_log_replay_check_allocations": "true", 2024-03-30T12:55:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_max_log_runway": "4194304", 2024-03-30T12:55:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_max_prefetch": "1048576", 2024-03-30T12:55:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_min_flush_size": "524288", 2024-03-30T12:55:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_min_log_runway": "1048576", 2024-03-30T12:55:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_replay_recovery": "false", 2024-03-30T12:55:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-03-30T12:55:39.723 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_shared_alloc_size": "65536", 2024-03-30T12:55:39.724 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_sync_write": "false", 2024-03-30T12:55:39.724 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-03-30T12:55:39.724 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-03-30T12:55:39.724 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-03-30T12:55:39.724 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_allocation_from_file": "true", 2024-03-30T12:55:39.724 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_allocator": "hybrid", 2024-03-30T12:55:39.724 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-03-30T12:55:39.724 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-03-30T12:55:39.724 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-03-30T12:55:39.724 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-03-30T12:55:39.724 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-03-30T12:55:39.724 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-03-30T12:55:39.724 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_blobid_prealloc": "10240", 2024-03-30T12:55:39.724 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_create": "true", 2024-03-30T12:55:39.724 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_db_create": "false", 2024-03-30T12:55:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_db_path": "", 2024-03-30T12:55:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_db_size": "0", 2024-03-30T12:55:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_path": "", 2024-03-30T12:55:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_preallocate_file": "false", 2024-03-30T12:55:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_size": "107374182400", 2024-03-30T12:55:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_wal_create": "false", 2024-03-30T12:55:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_wal_path": "", 2024-03-30T12:55:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_block_wal_size": "100663296", 2024-03-30T12:55:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_bluefs": "true", 2024-03-30T12:55:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-03-30T12:55:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_bluefs_env_mirror": "false", 2024-03-30T12:55:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-03-30T12:55:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-03-30T12:55:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-03-30T12:55:39.725 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-03-30T12:55:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-03-30T12:55:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-03-30T12:55:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_autotune": "true", 2024-03-30T12:55:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-03-30T12:55:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-03-30T12:55:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-03-30T12:55:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-03-30T12:55:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_size": "0", 2024-03-30T12:55:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-03-30T12:55:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-03-30T12:55:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-03-30T12:55:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-03-30T12:55:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_type": "2q", 2024-03-30T12:55:39.726 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-03-30T12:55:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_clone_cow": "true", 2024-03-30T12:55:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_compression_algorithm": "snappy", 2024-03-30T12:55:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_compression_max_blob_size": "0", 2024-03-30T12:55:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-03-30T12:55:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-03-30T12:55:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_compression_min_blob_size": "0", 2024-03-30T12:55:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-03-30T12:55:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-03-30T12:55:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_compression_mode": "none", 2024-03-30T12:55:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-03-30T12:55:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_csum_type": "crc32c", 2024-03-30T12:55:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-03-30T12:55:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_enforce_settings": "default", 2024-03-30T12:55:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_freelist": "false", 2024-03-30T12:55:39.727 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_fsck_abort": "false", 2024-03-30T12:55:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-03-30T12:55:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-03-30T12:55:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_inject_read_err": "false", 2024-03-30T12:55:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_legacy_omap": "false", 2024-03-30T12:55:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-03-30T12:55:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-03-30T12:55:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-03-30T12:55:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-03-30T12:55:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_prefill": "0.000000", 2024-03-30T12:55:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-03-30T12:55:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-03-30T12:55:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-03-30T12:55:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_small_allocations": "0", 2024-03-30T12:55:39.728 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-03-30T12:55:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_default_buffered_read": "true", 2024-03-30T12:55:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_default_buffered_write": "false", 2024-03-30T12:55:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_deferred_batch_ops": "0", 2024-03-30T12:55:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-03-30T12:55:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-03-30T12:55:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_elastic_shared_blobs": "true", 2024-03-30T12:55:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-03-30T12:55:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-03-30T12:55:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-03-30T12:55:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-03-30T12:55:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-03-30T12:55:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fail_eio": "false", 2024-03-30T12:55:39.729 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-03-30T12:55:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-03-30T12:55:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-03-30T12:55:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-03-30T12:55:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_on_mkfs": "true", 2024-03-30T12:55:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-03-30T12:55:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_on_mount": "false", 2024-03-30T12:55:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-03-30T12:55:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_on_umount": "false", 2024-03-30T12:55:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-03-30T12:55:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-03-30T12:55:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-03-30T12:55:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-03-30T12:55:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-03-30T12:55:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-03-30T12:55:39.730 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-03-30T12:55:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-03-30T12:55:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_ignore_data_csum": "false", 2024-03-30T12:55:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-03-30T12:55:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_kvbackend": "rocksdb", 2024-03-30T12:55:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-03-30T12:55:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-03-30T12:55:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_log_op_age": "5.000000", 2024-03-30T12:55:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_max_alloc_size": "0", 2024-03-30T12:55:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_max_blob_size": "0", 2024-03-30T12:55:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-03-30T12:55:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-03-30T12:55:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_max_defer_interval": "3.000000", 2024-03-30T12:55:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_max_deferred_txc": "32", 2024-03-30T12:55:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_min_alloc_size": "0", 2024-03-30T12:55:39.731 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-03-30T12:55:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-03-30T12:55:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_nid_prealloc": "1024", 2024-03-30T12:55:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_prefer_deferred_size": "0", 2024-03-30T12:55:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-03-30T12:55:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-03-30T12:55:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_qfsck_on_mount": "true", 2024-03-30T12:55:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_retry_disk_reads": "3", 2024-03-30T12:55:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_rocksdb_cf": "true", 2024-03-30T12:55:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-03-30T12:55:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-03-30T12:55:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_rocksdb_options_annex": "", 2024-03-30T12:55:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_spdk_coremask": "0x1", 2024-03-30T12:55:39.732 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_spdk_io_sleep": "5", 2024-03-30T12:55:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_spdk_max_io_completion": "0", 2024-03-30T12:55:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_spdk_mem": "512", 2024-03-30T12:55:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_sync_submit_transaction": "false", 2024-03-30T12:55:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_throttle_bytes": "67108864", 2024-03-30T12:55:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_throttle_cost_per_io": "0", 2024-03-30T12:55:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-03-30T12:55:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-03-30T12:55:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-03-30T12:55:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-03-30T12:55:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_tracing": "false", 2024-03-30T12:55:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-03-30T12:55:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-03-30T12:55:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_volume_selection_reserved": "0", 2024-03-30T12:55:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-03-30T12:55:39.733 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-03-30T12:55:39.734 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-03-30T12:55:39.734 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-03-30T12:55:39.734 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-03-30T12:55:39.734 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-03-30T12:55:39.734 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_zero_block_detection": "false", 2024-03-30T12:55:39.734 INFO:teuthology.orchestra.run.smithi067.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-03-30T12:55:39.734 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_action_update_interval": "2", 2024-03-30T12:55:39.734 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-03-30T12:55:39.734 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-03-30T12:55:39.734 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-03-30T12:55:39.734 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-03-30T12:55:39.734 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_mount_timeout": "10", 2024-03-30T12:55:39.734 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-03-30T12:55:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-03-30T12:55:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-03-30T12:55:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-03-30T12:55:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-03-30T12:55:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-03-30T12:55:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-03-30T12:55:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: "cephx_cluster_require_signatures": "false", 2024-03-30T12:55:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: "cephx_cluster_require_version": "2", 2024-03-30T12:55:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: "cephx_require_signatures": "false", 2024-03-30T12:55:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: "cephx_require_version": "2", 2024-03-30T12:55:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: "cephx_service_require_signatures": "false", 2024-03-30T12:55:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: "cephx_service_require_version": "2", 2024-03-30T12:55:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: "cephx_sign_messages": "true", 2024-03-30T12:55:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: "chdir": "", 2024-03-30T12:55:39.735 INFO:teuthology.orchestra.run.smithi067.stdout: "client_acl_type": "", 2024-03-30T12:55:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_asio_thread_count": "2", 2024-03-30T12:55:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_cache_mid": "0.750000", 2024-03-30T12:55:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_cache_size": "16384", 2024-03-30T12:55:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_caps_release_delay": "5", 2024-03-30T12:55:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_check_pool_perm": "true", 2024-03-30T12:55:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_collect_and_send_global_metrics": "false", 2024-03-30T12:55:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_debug_force_sync_read": "false", 2024-03-30T12:55:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_debug_getattr_caps": "false", 2024-03-30T12:55:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_debug_inject_tick_delay": "0", 2024-03-30T12:55:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-03-30T12:55:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_die_on_failed_remount": "false", 2024-03-30T12:55:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_dirsize_rbytes": "true", 2024-03-30T12:55:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_force_lazyio": "false", 2024-03-30T12:55:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_fs": "", 2024-03-30T12:55:39.736 INFO:teuthology.orchestra.run.smithi067.stdout: "client_inject_fixed_oldest_tid": "false", 2024-03-30T12:55:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: "client_inject_release_failure": "false", 2024-03-30T12:55:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: "client_max_inline_size": "4096", 2024-03-30T12:55:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: "client_max_retries_on_remount_failure": "5", 2024-03-30T12:55:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: "client_mds_namespace": "", 2024-03-30T12:55:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: "client_metadata": "", 2024-03-30T12:55:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: "client_mount_gid": "-1", 2024-03-30T12:55:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: "client_mount_timeout": "300", 2024-03-30T12:55:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: "client_mount_uid": "-1", 2024-03-30T12:55:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: "client_mountpoint": "/", 2024-03-30T12:55:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: "client_notify_timeout": "10", 2024-03-30T12:55:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: "client_oc": "true", 2024-03-30T12:55:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: "client_oc_max_dirty": "104857600", 2024-03-30T12:55:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: "client_oc_max_dirty_age": "5.000000", 2024-03-30T12:55:39.737 INFO:teuthology.orchestra.run.smithi067.stdout: "client_oc_max_objects": "1000", 2024-03-30T12:55:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: "client_oc_size": "209715200", 2024-03-30T12:55:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: "client_oc_target_dirty": "8388608", 2024-03-30T12:55:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: "client_permissions": "true", 2024-03-30T12:55:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: "client_quota": "true", 2024-03-30T12:55:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: "client_quota_df": "true", 2024-03-30T12:55:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: "client_readahead_max_bytes": "0", 2024-03-30T12:55:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: "client_readahead_max_periods": "4", 2024-03-30T12:55:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: "client_readahead_min": "131072", 2024-03-30T12:55:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: "client_reconnect_stale": "false", 2024-03-30T12:55:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: "client_shutdown_timeout": "30", 2024-03-30T12:55:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: "client_snapdir": ".snap", 2024-03-30T12:55:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: "client_tick_interval": "1", 2024-03-30T12:55:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: "client_trace": "", 2024-03-30T12:55:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: "client_try_dentry_invalidate": "false", 2024-03-30T12:55:39.738 INFO:teuthology.orchestra.run.smithi067.stdout: "client_use_faked_inos": "false", 2024-03-30T12:55:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: "client_use_random_mds": "false", 2024-03-30T12:55:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: "clog_to_graylog": "false", 2024-03-30T12:55:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-03-30T12:55:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: "clog_to_graylog_port": "12201", 2024-03-30T12:55:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: "clog_to_monitors": "default=true", 2024-03-30T12:55:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: "clog_to_syslog": "false", 2024-03-30T12:55:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-03-30T12:55:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: "clog_to_syslog_level": "info", 2024-03-30T12:55:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_addr": "-", 2024-03-30T12:55:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_network": "", 2024-03-30T12:55:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_network_interface": "", 2024-03-30T12:55:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: "compressor_zlib_isal": "false", 2024-03-30T12:55:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: "compressor_zlib_level": "5", 2024-03-30T12:55:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: "compressor_zlib_winsize": "-15", 2024-03-30T12:55:39.739 INFO:teuthology.orchestra.run.smithi067.stdout: "compressor_zstd_level": "1", 2024-03-30T12:55:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-03-30T12:55:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-03-30T12:55:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: "crimson_alien_op_num_threads": "6", 2024-03-30T12:55:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: "crimson_alien_thread_cpu_cores": "", 2024-03-30T12:55:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: "crimson_osd_obc_lru_size": "512", 2024-03-30T12:55:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-03-30T12:55:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: "crimson_seastar_cpu_cores": "", 2024-03-30T12:55:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_location": "", 2024-03-30T12:55:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_location_hook": "", 2024-03-30T12:55:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: "crush_location_hook_timeout": "10", 2024-03-30T12:55:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: "daemonize": "false", 2024-03-30T12:55:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: "daos_pool": "tank", 2024-03-30T12:55:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-03-30T12:55:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-03-30T12:55:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: "dbstore_db_name_prefix": "dbstore", 2024-03-30T12:55:39.741 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_alienstore": "0/5", 2024-03-30T12:55:39.741 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_allow_any_pool_priority": "false", 2024-03-30T12:55:39.741 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_asok": "1/5", 2024-03-30T12:55:39.741 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_asok_assert_abort": "false", 2024-03-30T12:55:39.741 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_asserts_on_shutdown": "true", 2024-03-30T12:55:39.741 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_auth": "20/20", 2024-03-30T12:55:39.741 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_bdev": "1/3", 2024-03-30T12:55:39.741 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_bluefs": "1/5", 2024-03-30T12:55:39.741 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_bluestore": "1/5", 2024-03-30T12:55:39.741 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_buffer": "0/1", 2024-03-30T12:55:39.741 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_ceph_exporter": "1/5", 2024-03-30T12:55:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_cephfs_mirror": "0/5", 2024-03-30T12:55:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_cephsqlite": "0/5", 2024-03-30T12:55:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_client": "0/5", 2024-03-30T12:55:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_compressor": "1/5", 2024-03-30T12:55:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_context": "0/1", 2024-03-30T12:55:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_crimson_interrupt": "0/5", 2024-03-30T12:55:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_crush": "1/1", 2024-03-30T12:55:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_crypto": "1/5", 2024-03-30T12:55:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_cyanstore": "0/5", 2024-03-30T12:55:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_deliberately_leak_memory": "false", 2024-03-30T12:55:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_disable_randomized_ping": "false", 2024-03-30T12:55:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_dpdk": "1/5", 2024-03-30T12:55:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_eventtrace": "1/5", 2024-03-30T12:55:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_filer": "0/1", 2024-03-30T12:55:39.742 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_filestore": "1/3", 2024-03-30T12:55:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_finisher": "1/1", 2024-03-30T12:55:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_fuse": "1/5", 2024-03-30T12:55:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_heartbeat_testing_span": "0", 2024-03-30T12:55:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_heartbeatmap": "1/5", 2024-03-30T12:55:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_immutable_obj_cache": "0/5", 2024-03-30T12:55:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_javaclient": "1/5", 2024-03-30T12:55:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_journal": "1/3", 2024-03-30T12:55:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_journaler": "0/5", 2024-03-30T12:55:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_kstore": "1/5", 2024-03-30T12:55:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_lockdep": "0/1", 2024-03-30T12:55:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mclock": "1/5", 2024-03-30T12:55:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mds": "1/5", 2024-03-30T12:55:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mds_balancer": "1/5", 2024-03-30T12:55:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mds_locker": "1/5", 2024-03-30T12:55:39.743 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mds_log": "1/5", 2024-03-30T12:55:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mds_log_expire": "1/5", 2024-03-30T12:55:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mds_migrator": "1/5", 2024-03-30T12:55:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_memstore": "1/5", 2024-03-30T12:55:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mgr": "2/5", 2024-03-30T12:55:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mgrc": "1/5", 2024-03-30T12:55:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_mon": "20/20", 2024-03-30T12:55:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_monc": "0/10", 2024-03-30T12:55:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_ms": "1/1", 2024-03-30T12:55:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_none": "0/5", 2024-03-30T12:55:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_objclass": "0/5", 2024-03-30T12:55:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_objectcacher": "0/5", 2024-03-30T12:55:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_objecter": "0/1", 2024-03-30T12:55:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_optracker": "0/5", 2024-03-30T12:55:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_osd": "1/5", 2024-03-30T12:55:39.744 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_paxos": "20/20", 2024-03-30T12:55:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_perfcounter": "1/5", 2024-03-30T12:55:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_prioritycache": "1/5", 2024-03-30T12:55:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rados": "0/5", 2024-03-30T12:55:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rbd": "0/5", 2024-03-30T12:55:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rbd_mirror": "0/5", 2024-03-30T12:55:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rbd_pwl": "0/5", 2024-03-30T12:55:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rbd_replay": "0/5", 2024-03-30T12:55:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_refs": "0/0", 2024-03-30T12:55:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_reserver": "1/1", 2024-03-30T12:55:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rgw": "1/5", 2024-03-30T12:55:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rgw_access": "1/5", 2024-03-30T12:55:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rgw_datacache": "1/5", 2024-03-30T12:55:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rgw_dbstore": "1/5", 2024-03-30T12:55:39.745 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rgw_flight": "1/5", 2024-03-30T12:55:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rgw_lifecycle": "1/5", 2024-03-30T12:55:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rgw_sync": "1/5", 2024-03-30T12:55:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_rocksdb": "4/5", 2024-03-30T12:55:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore": "0/5", 2024-03-30T12:55:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_backref": "0/5", 2024-03-30T12:55:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_cache": "0/5", 2024-03-30T12:55:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_cleaner": "0/5", 2024-03-30T12:55:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_device": "0/5", 2024-03-30T12:55:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_epm": "0/5", 2024-03-30T12:55:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-03-30T12:55:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_journal": "0/5", 2024-03-30T12:55:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_lba": "0/5", 2024-03-30T12:55:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_odata": "0/5", 2024-03-30T12:55:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_omap": "0/5", 2024-03-30T12:55:39.746 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_onode": "0/5", 2024-03-30T12:55:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_t": "0/5", 2024-03-30T12:55:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_seastore_tm": "0/5", 2024-03-30T12:55:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_striper": "0/1", 2024-03-30T12:55:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_test": "0/5", 2024-03-30T12:55:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_throttle": "1/1", 2024-03-30T12:55:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_timer": "0/1", 2024-03-30T12:55:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_tp": "0/5", 2024-03-30T12:55:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: "debug_trace": "1/5", 2024-03-30T12:55:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: "defer_client_eviction_on_laggy_osds": "true", 2024-03-30T12:55:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: "device_failure_prediction_mode": "none", 2024-03-30T12:55:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "*", 2024-03-30T12:55:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2024-03-30T12:55:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: "err_to_graylog": "false", 2024-03-30T12:55:39.747 INFO:teuthology.orchestra.run.smithi067.stdout: "err_to_journald": "false", 2024-03-30T12:55:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: "err_to_stderr": "true", 2024-03-30T12:55:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: "err_to_syslog": "false", 2024-03-30T12:55:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: "event_tracing": "false", 2024-03-30T12:55:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: "exporter_addr": "0.0.0.0", 2024-03-30T12:55:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: "exporter_http_port": "9926", 2024-03-30T12:55:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: "exporter_prio_limit": "5", 2024-03-30T12:55:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-03-30T12:55:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: "exporter_sort_metrics": "true", 2024-03-30T12:55:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: "exporter_stats_period": "5", 2024-03-30T12:55:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: "fake_statfs_for_testing": "0", 2024-03-30T12:55:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: "fatal_signal_handlers": "true", 2024-03-30T12:55:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: "filer_max_purge_ops": "10", 2024-03-30T12:55:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: "filer_max_truncate_ops": "128", 2024-03-30T12:55:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_apply_finisher_threads": "1", 2024-03-30T12:55:39.748 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_blackhole": "false", 2024-03-30T12:55:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_btrfs_clone_range": "true", 2024-03-30T12:55:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_btrfs_snap": "true", 2024-03-30T12:55:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_caller_concurrency": "10", 2024-03-30T12:55:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_collect_device_partition_information": "true", 2024-03-30T12:55:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_commit_timeout": "600.000000", 2024-03-30T12:55:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_debug_inject_read_err": "false", 2024-03-30T12:55:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_debug_omap_check": "false", 2024-03-30T12:55:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_debug_random_read_err": "0.000000", 2024-03-30T12:55:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_debug_verify_split": "false", 2024-03-30T12:55:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_dump_file": "", 2024-03-30T12:55:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-03-30T12:55:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-03-30T12:55:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_fadvise": "true", 2024-03-30T12:55:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_fail_eio": "true", 2024-03-30T12:55:39.749 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_fd_cache_shards": "16", 2024-03-30T12:55:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_fd_cache_size": "128", 2024-03-30T12:55:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_fiemap": "false", 2024-03-30T12:55:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_fiemap_threshold": "4096", 2024-03-30T12:55:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-03-30T12:55:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_index_retry_probability": "0.000000", 2024-03-30T12:55:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_inject_stall": "0", 2024-03-30T12:55:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_journal_parallel": "false", 2024-03-30T12:55:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_journal_trailing": "false", 2024-03-30T12:55:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_journal_writeahead": "false", 2024-03-30T12:55:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_kill_at": "0", 2024-03-30T12:55:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-03-30T12:55:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_inline_xattr_size": "0", 2024-03-30T12:55:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-03-30T12:55:39.750 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-03-30T12:55:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-03-30T12:55:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_inline_xattrs": "0", 2024-03-30T12:55:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-03-30T12:55:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_inline_xattrs_other": "2", 2024-03-30T12:55:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-03-30T12:55:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_sync_interval": "5.000000", 2024-03-30T12:55:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_xattr_value_size": "0", 2024-03-30T12:55:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-03-30T12:55:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-03-30T12:55:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-03-30T12:55:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_merge_threshold": "-10", 2024-03-30T12:55:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_min_sync_interval": "0.010000", 2024-03-30T12:55:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_odsync_write": "false", 2024-03-30T12:55:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_omap_backend": "rocksdb", 2024-03-30T12:55:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_omap_backend_path": "", 2024-03-30T12:55:39.751 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_omap_header_cache_size": "1024", 2024-03-30T12:55:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_ondisk_finisher_threads": "1", 2024-03-30T12:55:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-03-30T12:55:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_op_thread_timeout": "60", 2024-03-30T12:55:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_op_threads": "2", 2024-03-30T12:55:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_punch_hole": "false", 2024-03-30T12:55:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-03-30T12:55:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-03-30T12:55:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-03-30T12:55:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-03-30T12:55:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-03-30T12:55:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_max_bytes": "104857600", 2024-03-30T12:55:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-03-30T12:55:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-03-30T12:55:39.752 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-03-30T12:55:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_queue_max_ops": "50", 2024-03-30T12:55:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-03-30T12:55:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_seek_data_hole": "false", 2024-03-30T12:55:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_sloppy_crc": "false", 2024-03-30T12:55:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-03-30T12:55:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_splice": "false", 2024-03-30T12:55:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_split_multiple": "2", 2024-03-30T12:55:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_split_rand_factor": "20", 2024-03-30T12:55:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_update_to": "1000", 2024-03-30T12:55:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-03-30T12:55:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-03-30T12:55:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-03-30T12:55:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-03-30T12:55:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-03-30T12:55:39.753 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-03-30T12:55:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_enable": "true", 2024-03-30T12:55:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-03-30T12:55:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-03-30T12:55:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-03-30T12:55:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-03-30T12:55:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-03-30T12:55:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-03-30T12:55:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_xfs_extsize": "false", 2024-03-30T12:55:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: "filestore_zfs_snap": "false", 2024-03-30T12:55:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: "fio_dir": "/tmp/fio", 2024-03-30T12:55:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: "fsid": "0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a", 2024-03-30T12:55:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_allow_other": "true", 2024-03-30T12:55:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_atomic_o_trunc": "true", 2024-03-30T12:55:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_big_writes": "true", 2024-03-30T12:55:39.754 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_debug": "false", 2024-03-30T12:55:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_default_permissions": "false", 2024-03-30T12:55:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_disable_pagecache": "false", 2024-03-30T12:55:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_max_write": "0", 2024-03-30T12:55:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_multithreaded": "true", 2024-03-30T12:55:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_require_active_mds": "true", 2024-03-30T12:55:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_set_user_groups": "true", 2024-03-30T12:55:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_splice_move": "true", 2024-03-30T12:55:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_splice_read": "true", 2024-03-30T12:55:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_splice_write": "true", 2024-03-30T12:55:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_syncfs_on_mksnap": "true", 2024-03-30T12:55:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: "fuse_use_invalidate_cb": "true", 2024-03-30T12:55:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-03-30T12:55:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: "gss_target_name": "ceph", 2024-03-30T12:55:39.755 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_file": "", 2024-03-30T12:55:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_inject_failure": "0", 2024-03-30T12:55:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: "heartbeat_interval": "5", 2024-03-30T12:55:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: "host": "smithi067", 2024-03-30T12:55:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-03-30T12:55:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-03-30T12:55:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_max_size": "1073741824", 2024-03-30T12:55:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-03-30T12:55:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-03-30T12:55:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-03-30T12:55:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-03-30T12:55:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-03-30T12:55:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-03-30T12:55:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-03-30T12:55:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-03-30T12:55:39.756 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-03-30T12:55:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: "immutable_object_cache_watermark": "0.900000", 2024-03-30T12:55:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: "inject_early_sigterm": "false", 2024-03-30T12:55:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: "jaeger_agent_port": "6799", 2024-03-30T12:55:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: "jaeger_tracing_enable": "false", 2024-03-30T12:55:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_aio": "true", 2024-03-30T12:55:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_align_min_size": "65536", 2024-03-30T12:55:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_block_align": "true", 2024-03-30T12:55:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_block_size": "4096", 2024-03-30T12:55:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_dio": "true", 2024-03-30T12:55:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_discard": "false", 2024-03-30T12:55:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_force_aio": "false", 2024-03-30T12:55:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_ignore_corruption": "false", 2024-03-30T12:55:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_max_write_bytes": "10485760", 2024-03-30T12:55:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_max_write_entries": "100", 2024-03-30T12:55:39.757 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_replay_from": "0", 2024-03-30T12:55:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_throttle_high_multiple": "0.000000", 2024-03-30T12:55:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-03-30T12:55:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-03-30T12:55:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_throttle_max_multiple": "0.000000", 2024-03-30T12:55:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_write_header_frequency": "0", 2024-03-30T12:55:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: "journal_zero_on_create": "false", 2024-03-30T12:55:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: "journaler_prefetch_periods": "10", 2024-03-30T12:55:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: "journaler_prezero_periods": "5", 2024-03-30T12:55:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: "journaler_write_head_interval": "15", 2024-03-30T12:55:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "", 2024-03-30T12:55:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: "keyfile": "", 2024-03-30T12:55:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-03-30T12:55:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_backend": "rocksdb", 2024-03-30T12:55:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_default_stripe_size": "65536", 2024-03-30T12:55:39.758 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_fsck_on_mount": "false", 2024-03-30T12:55:39.759 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_fsck_on_mount_deep": "true", 2024-03-30T12:55:39.759 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_max_bytes": "67108864", 2024-03-30T12:55:39.759 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_max_ops": "512", 2024-03-30T12:55:39.759 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_nid_prealloc": "1024", 2024-03-30T12:55:39.759 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_onode_map_size": "1024", 2024-03-30T12:55:39.759 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-03-30T12:55:39.759 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_sync_submit_transaction": "false", 2024-03-30T12:55:39.759 INFO:teuthology.orchestra.run.smithi067.stdout: "kstore_sync_transaction": "false", 2024-03-30T12:55:39.759 INFO:teuthology.orchestra.run.smithi067.stdout: "librados_thread_count": "2", 2024-03-30T12:55:39.759 INFO:teuthology.orchestra.run.smithi067.stdout: "lockdep": "false", 2024-03-30T12:55:39.759 INFO:teuthology.orchestra.run.smithi067.stdout: "lockdep_force_backtrace": "false", 2024-03-30T12:55:39.759 INFO:teuthology.orchestra.run.smithi067.stdout: "log_coarse_timestamps": "true", 2024-03-30T12:55:39.759 INFO:teuthology.orchestra.run.smithi067.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-03-30T12:55:39.759 INFO:teuthology.orchestra.run.smithi067.stdout: "log_flush_on_exit": "false", 2024-03-30T12:55:39.759 INFO:teuthology.orchestra.run.smithi067.stdout: "log_graylog_host": "127.0.0.1", 2024-03-30T12:55:39.760 INFO:teuthology.orchestra.run.smithi067.stdout: "log_graylog_port": "12201", 2024-03-30T12:55:39.760 INFO:teuthology.orchestra.run.smithi067.stdout: "log_max_new": "1000", 2024-03-30T12:55:39.760 INFO:teuthology.orchestra.run.smithi067.stdout: "log_max_recent": "10000", 2024-03-30T12:55:39.760 INFO:teuthology.orchestra.run.smithi067.stdout: "log_stderr_prefix": "", 2024-03-30T12:55:39.760 INFO:teuthology.orchestra.run.smithi067.stdout: "log_stop_at_utilization": "0.970000", 2024-03-30T12:55:39.760 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_file": "true", 2024-03-30T12:55:39.760 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_graylog": "false", 2024-03-30T12:55:39.760 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_journald": "false", 2024-03-30T12:55:39.760 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_stderr": "false", 2024-03-30T12:55:39.760 INFO:teuthology.orchestra.run.smithi067.stdout: "log_to_syslog": "false", 2024-03-30T12:55:39.760 INFO:teuthology.orchestra.run.smithi067.stdout: "max_rotating_auth_attempts": "10", 2024-03-30T12:55:39.760 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-03-30T12:55:39.760 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_action_on_write_error": "1", 2024-03-30T12:55:39.761 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_alternate_name_max": "8192", 2024-03-30T12:55:39.761 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_asio_thread_count": "2", 2024-03-30T12:55:39.761 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_export_pin": "true", 2024-03-30T12:55:39.761 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_fragment_dirs": "true", 2024-03-30T12:55:39.761 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-03-30T12:55:39.761 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_fragment_interval": "5", 2024-03-30T12:55:39.761 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_fragment_size_max": "100000", 2024-03-30T12:55:39.761 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_idle_threshold": "0.000000", 2024-03-30T12:55:39.761 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_interval": "10", 2024-03-30T12:55:39.761 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_max": "-1", 2024-03-30T12:55:39.761 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_max_until": "-1", 2024-03-30T12:55:39.761 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_merge_size": "50", 2024-03-30T12:55:39.761 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_midchunk": "0.300000", 2024-03-30T12:55:39.761 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_min_rebalance": "0.100000", 2024-03-30T12:55:39.761 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_min_start": "0.200000", 2024-03-30T12:55:39.762 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_minchunk": "0.001000", 2024-03-30T12:55:39.762 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_mode": "0", 2024-03-30T12:55:39.762 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_need_max": "1.200000", 2024-03-30T12:55:39.762 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_need_min": "0.800000", 2024-03-30T12:55:39.762 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_overload_epochs": "2", 2024-03-30T12:55:39.762 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-03-30T12:55:39.762 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_sample_interval": "3.000000", 2024-03-30T12:55:39.762 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_split_bits": "3", 2024-03-30T12:55:39.762 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_split_rd": "25000.000000", 2024-03-30T12:55:39.762 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_split_size": "10000", 2024-03-30T12:55:39.762 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_split_wr": "10000.000000", 2024-03-30T12:55:39.762 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_target_decay": "10.000000", 2024-03-30T12:55:39.762 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-03-30T12:55:39.762 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_beacon_grace": "15.000000", 2024-03-30T12:55:39.763 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_beacon_interval": "4.000000", 2024-03-30T12:55:39.763 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_beacon_mon_down_grace": "60", 2024-03-30T12:55:39.763 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache_memory_limit": "4294967296", 2024-03-30T12:55:39.763 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache_mid": "0.700000", 2024-03-30T12:55:39.763 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache_release_free_interval": "10", 2024-03-30T12:55:39.763 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache_reservation": "0.050000", 2024-03-30T12:55:39.763 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-03-30T12:55:39.763 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache_trim_interval": "1", 2024-03-30T12:55:39.763 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache_trim_threshold": "262144", 2024-03-30T12:55:39.763 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-03-30T12:55:39.763 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-03-30T12:55:39.763 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_client_delegate_inos_pct": "50", 2024-03-30T12:55:39.763 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_client_prealloc_inos": "1000", 2024-03-30T12:55:39.763 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-03-30T12:55:39.763 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_connect_bootstrapping": "false", 2024-03-30T12:55:39.763 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_damage_table_max_entries": "10000", 2024-03-30T12:55:39.764 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-03-30T12:55:39.764 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_debug_auth_pins": "false", 2024-03-30T12:55:39.764 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_debug_frag": "false", 2024-03-30T12:55:39.764 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_debug_scatterstat": "false", 2024-03-30T12:55:39.764 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_debug_subtrees": "false", 2024-03-30T12:55:39.764 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_decay_halflife": "5.000000", 2024-03-30T12:55:39.764 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_default_dir_hash": "2", 2024-03-30T12:55:39.764 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_defer_session_stale": "true", 2024-03-30T12:55:39.764 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_deny_all_reconnect": "false", 2024-03-30T12:55:39.764 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_dir_keys_per_op": "16384", 2024-03-30T12:55:39.764 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_dir_max_commit_size": "10", 2024-03-30T12:55:39.764 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_dir_max_entries": "0", 2024-03-30T12:55:39.764 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_dir_prefetch": "true", 2024-03-30T12:55:39.764 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_dirstat_min_interval": "1.000000", 2024-03-30T12:55:39.765 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_dump_cache_after_rejoin": "false", 2024-03-30T12:55:39.765 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_dump_cache_on_map": "false", 2024-03-30T12:55:39.765 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_dump_cache_threshold_file": "0", 2024-03-30T12:55:39.765 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-03-30T12:55:39.765 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_early_reply": "true", 2024-03-30T12:55:39.765 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_enable_op_tracker": "true", 2024-03-30T12:55:39.765 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_enforce_unique_name": "true", 2024-03-30T12:55:39.765 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_export_ephemeral_distributed": "true", 2024-03-30T12:55:39.765 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-03-30T12:55:39.765 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_export_ephemeral_random": "true", 2024-03-30T12:55:39.765 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-03-30T12:55:39.765 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-03-30T12:55:39.765 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_forward_all_requests_to_auth": "false", 2024-03-30T12:55:39.765 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-03-30T12:55:39.766 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-03-30T12:55:39.766 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-03-30T12:55:39.766 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-03-30T12:55:39.766 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_health_cache_threshold": "1.500000", 2024-03-30T12:55:39.766 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_health_summarize_threshold": "10", 2024-03-30T12:55:39.766 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_heartbeat_grace": "15.000000", 2024-03-30T12:55:39.766 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_heartbeat_reset_grace": "1000", 2024-03-30T12:55:39.766 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_inject_health_dummy": "false", 2024-03-30T12:55:39.766 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-03-30T12:55:39.766 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_inject_migrator_session_race": "false", 2024-03-30T12:55:39.766 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-03-30T12:55:39.766 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-03-30T12:55:39.766 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-03-30T12:55:39.766 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_join_fs": "", 2024-03-30T12:55:39.766 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_journal_format": "1", 2024-03-30T12:55:39.767 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_create_at": "0", 2024-03-30T12:55:39.767 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_export_at": "0", 2024-03-30T12:55:39.767 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_import_at": "0", 2024-03-30T12:55:39.767 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_journal_at": "0", 2024-03-30T12:55:39.767 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_journal_expire_at": "0", 2024-03-30T12:55:39.767 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_journal_replay_at": "0", 2024-03-30T12:55:39.767 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_link_at": "0", 2024-03-30T12:55:39.767 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_mdstable_at": "0", 2024-03-30T12:55:39.767 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_openc_at": "0", 2024-03-30T12:55:39.767 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_rename_at": "0", 2024-03-30T12:55:39.767 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_shutdown_at": "0", 2024-03-30T12:55:39.767 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-03-30T12:55:39.767 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_event_large_threshold": "524288", 2024-03-30T12:55:39.767 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_events_per_segment": "1024", 2024-03-30T12:55:39.768 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_major_segment_event_ratio": "12", 2024-03-30T12:55:39.768 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_max_events": "-1", 2024-03-30T12:55:39.768 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_max_segments": "128", 2024-03-30T12:55:39.768 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_pause": "false", 2024-03-30T12:55:39.768 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_segment_size": "0", 2024-03-30T12:55:39.768 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_skip_corrupt_events": "false", 2024-03-30T12:55:39.768 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_skip_unbounded_events": "false", 2024-03-30T12:55:39.768 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-03-30T12:55:39.768 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_trim_threshold": "128", 2024-03-30T12:55:39.768 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-03-30T12:55:39.768 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log_warn_factor": "2.000000", 2024-03-30T12:55:39.768 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_caps_per_client": "1048576", 2024-03-30T12:55:39.768 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_completed_flushes": "100000", 2024-03-30T12:55:39.768 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_completed_requests": "100000", 2024-03-30T12:55:39.768 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_export_size": "20971520", 2024-03-30T12:55:39.769 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_file_recover": "32", 2024-03-30T12:55:39.769 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_purge_files": "64", 2024-03-30T12:55:39.769 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_purge_ops": "8192", 2024-03-30T12:55:39.769 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-03-30T12:55:39.769 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-03-30T12:55:39.769 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_max_snaps_per_dir": "100", 2024-03-30T12:55:39.769 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_metrics_update_interval": "2", 2024-03-30T12:55:39.769 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_min_caps_per_client": "100", 2024-03-30T12:55:39.769 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_min_caps_working_set": "10000", 2024-03-30T12:55:39.769 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-03-30T12:55:39.769 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_numa_node": "-1", 2024-03-30T12:55:39.769 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-03-30T12:55:39.769 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_op_complaint_time": "30.000000", 2024-03-30T12:55:39.769 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_op_history_duration": "600", 2024-03-30T12:55:39.769 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_op_history_size": "20", 2024-03-30T12:55:39.770 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_op_history_slow_op_size": "20", 2024-03-30T12:55:39.770 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_op_history_slow_op_threshold": "10", 2024-03-30T12:55:39.770 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_op_log_threshold": "5", 2024-03-30T12:55:39.770 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_ping_grace": "15", 2024-03-30T12:55:39.770 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_ping_interval": "5", 2024-03-30T12:55:39.770 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-03-30T12:55:39.770 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-03-30T12:55:39.770 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_recall_max_caps": "30000", 2024-03-30T12:55:39.770 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-03-30T12:55:39.770 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_recall_max_decay_threshold": "131072", 2024-03-30T12:55:39.770 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-03-30T12:55:39.770 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_recall_warning_threshold": "262144", 2024-03-30T12:55:39.770 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_reconnect_timeout": "45.000000", 2024-03-30T12:55:39.770 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_replay_interval": "1.000000", 2024-03-30T12:55:39.771 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-03-30T12:55:39.771 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-03-30T12:55:39.771 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_root_ino_gid": "0", 2024-03-30T12:55:39.771 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_root_ino_uid": "0", 2024-03-30T12:55:39.771 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-03-30T12:55:39.771 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_session_blocklist_on_evict": "true", 2024-03-30T12:55:39.771 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_session_blocklist_on_timeout": "true", 2024-03-30T12:55:39.771 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-03-30T12:55:39.771 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-03-30T12:55:39.771 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-03-30T12:55:39.771 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-03-30T12:55:39.771 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-03-30T12:55:39.771 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_session_metadata_threshold": "16777216", 2024-03-30T12:55:39.771 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-03-30T12:55:39.771 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_shutdown_check": "0", 2024-03-30T12:55:39.772 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_skip_ino": "0", 2024-03-30T12:55:39.772 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_sleep_rank_change": "0.000000", 2024-03-30T12:55:39.772 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_snap_max_uid": "4294967294", 2024-03-30T12:55:39.772 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_snap_min_uid": "0", 2024-03-30T12:55:39.772 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_snap_rstat": "false", 2024-03-30T12:55:39.772 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_standby_replay_damaged": "false", 2024-03-30T12:55:39.772 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_symlink_recovery": "true", 2024-03-30T12:55:39.772 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_task_status_update_interval": "2.000000", 2024-03-30T12:55:39.772 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_thrash_exports": "0", 2024-03-30T12:55:39.772 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_thrash_fragments": "0", 2024-03-30T12:55:39.772 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_tick_interval": "5.000000", 2024-03-30T12:55:39.772 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_valgrind_exit": "false", 2024-03-30T12:55:39.772 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_verify_backtrace": "1", 2024-03-30T12:55:39.772 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_verify_scatter": "false", 2024-03-30T12:55:39.773 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_wipe_ino_prealloc": "false", 2024-03-30T12:55:39.773 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_wipe_sessions": "false", 2024-03-30T12:55:39.773 INFO:teuthology.orchestra.run.smithi067.stdout: "mempool_debug": "false", 2024-03-30T12:55:39.773 INFO:teuthology.orchestra.run.smithi067.stdout: "memstore_debug_omit_block_device_write": "false", 2024-03-30T12:55:39.773 INFO:teuthology.orchestra.run.smithi067.stdout: "memstore_device_bytes": "1073741824", 2024-03-30T12:55:39.773 INFO:teuthology.orchestra.run.smithi067.stdout: "memstore_page_set": "false", 2024-03-30T12:55:39.773 INFO:teuthology.orchestra.run.smithi067.stdout: "memstore_page_size": "65536", 2024-03-30T12:55:39.773 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_client_bytes": "134217728", 2024-03-30T12:55:39.773 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_client_messages": "512", 2024-03-30T12:55:39.773 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-03-30T12:55:39.773 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_connect_retry_interval": "1.000000", 2024-03-30T12:55:39.773 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-03-30T12:55:39.773 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-03-30T12:55:39.773 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_disabled_modules": "", 2024-03-30T12:55:39.773 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-03-30T12:55:39.774 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_max_pg_num_change": "128", 2024-03-30T12:55:39.774 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_mds_bytes": "134217728", 2024-03-30T12:55:39.774 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_mds_messages": "128", 2024-03-30T12:55:39.774 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-03-30T12:55:39.774 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_mon_bytes": "134217728", 2024-03-30T12:55:39.774 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_mon_messages": "128", 2024-03-30T12:55:39.774 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_osd_bytes": "536870912", 2024-03-30T12:55:39.774 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_osd_messages": "8192", 2024-03-30T12:55:39.774 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_pool": "true", 2024-03-30T12:55:39.774 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_service_beacon_grace": "60.000000", 2024-03-30T12:55:39.774 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_standby_modules": "true", 2024-03-30T12:55:39.774 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_stats_period": "5", 2024-03-30T12:55:39.774 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_stats_threshold": "5", 2024-03-30T12:55:39.774 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_tick_period": "2", 2024-03-30T12:55:39.774 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-03-30T12:55:39.775 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_accept_timeout_factor": "2.000000", 2024-03-30T12:55:39.775 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_allow_pool_delete": "true", 2024-03-30T12:55:39.775 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_allow_pool_size_one": "true", 2024-03-30T12:55:39.775 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_auth_validate_all_caps": "true", 2024-03-30T12:55:39.775 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-03-30T12:55:39.775 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-03-30T12:55:39.775 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_bytes": "104857600", 2024-03-30T12:55:39.775 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_directed_command_retry": "5", 2024-03-30T12:55:39.775 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_hunt_interval": "3.000000", 2024-03-30T12:55:39.775 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-03-30T12:55:39.775 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-03-30T12:55:39.775 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-03-30T12:55:39.775 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_hunt_parallel": "3", 2024-03-30T12:55:39.775 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_log_interval": "1.000000", 2024-03-30T12:55:39.776 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-03-30T12:55:39.776 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_ping_interval": "10.000000", 2024-03-30T12:55:39.776 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_ping_timeout": "30.000000", 2024-03-30T12:55:39.776 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_client_target_rank": "-1", 2024-03-30T12:55:39.776 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_clock_drift_allowed": "1.000000", 2024-03-30T12:55:39.776 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-03-30T12:55:39.776 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-03-30T12:55:39.776 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_file_level": "debug", 2024-03-30T12:55:39.776 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_to_file": "true", 2024-03-30T12:55:39.776 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_to_graylog": "false", 2024-03-30T12:55:39.776 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-03-30T12:55:39.776 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-03-30T12:55:39.776 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_to_journald": "false", 2024-03-30T12:55:39.776 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_to_stderr": "false", 2024-03-30T12:55:39.776 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-03-30T12:55:39.777 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-03-30T12:55:39.777 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-03-30T12:55:39.777 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_compact_on_bootstrap": "false", 2024-03-30T12:55:39.777 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_compact_on_start": "false", 2024-03-30T12:55:39.777 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_compact_on_trim": "true", 2024-03-30T12:55:39.777 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_con_tracker_persist_interval": "10", 2024-03-30T12:55:39.777 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_con_tracker_score_halflife": "43200", 2024-03-30T12:55:39.777 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_config_key_max_entry_size": "65536", 2024-03-30T12:55:39.777 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_cpu_threads": "4", 2024-03-30T12:55:39.777 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_crush_min_required_version": "hammer", 2024-03-30T12:55:39.777 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_daemon_bytes": "419430400", 2024-03-30T12:55:39.777 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-03-30T12:55:39.777 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_data_avail_crit": "5", 2024-03-30T12:55:39.777 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_data_avail_warn": "5", 2024-03-30T12:55:39.777 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_data_size_warn": "16106127360", 2024-03-30T12:55:39.778 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_block_osdmap_trim": "false", 2024-03-30T12:55:39.778 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-03-30T12:55:39.778 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_dump_json": "false", 2024-03-30T12:55:39.778 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-03-30T12:55:39.778 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_dump_transactions": "false", 2024-03-30T12:55:39.778 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_extra_checks": "false", 2024-03-30T12:55:39.778 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-03-30T12:55:39.778 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-03-30T12:55:39.778 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_no_require_reef": "false", 2024-03-30T12:55:39.778 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_no_require_squid": "false", 2024-03-30T12:55:39.778 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-03-30T12:55:39.778 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_delta_reset_interval": "10.000000", 2024-03-30T12:55:39.778 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_dns_srv_name": "ceph-mon", 2024-03-30T12:55:39.778 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_down_mkfs_grace": "120", 2024-03-30T12:55:39.779 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_election_default_strategy": "3", 2024-03-30T12:55:39.779 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_election_timeout": "5.000000", 2024-03-30T12:55:39.779 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-03-30T12:55:39.779 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_elector_ping_divisor": "2", 2024-03-30T12:55:39.779 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_elector_ping_timeout": "2.000000", 2024-03-30T12:55:39.779 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_enable_op_tracker": "true", 2024-03-30T12:55:39.779 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_fake_pool_delete": "false", 2024-03-30T12:55:39.779 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_force_quorum_join": "false", 2024-03-30T12:55:39.779 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_globalid_prealloc": "10000", 2024-03-30T12:55:39.779 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_health_detail_to_clog": "false", 2024-03-30T12:55:39.779 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_health_log_update_period": "5", 2024-03-30T12:55:39.779 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_health_max_detail": "50", 2024-03-30T12:55:39.779 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_health_to_clog": "true", 2024-03-30T12:55:39.779 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_health_to_clog_interval": "600", 2024-03-30T12:55:39.779 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-03-30T12:55:39.780 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_host": "172.21.15.67,[v2:172.21.15.67:3301,v1:172.21.15.67:6790],172.21.15.82", 2024-03-30T12:55:39.780 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_host_override": "", 2024-03-30T12:55:39.780 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_initial_members": "", 2024-03-30T12:55:39.780 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-03-30T12:55:39.780 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-03-30T12:55:39.780 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-03-30T12:55:39.780 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-03-30T12:55:39.780 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_keyvaluedb": "rocksdb", 2024-03-30T12:55:39.780 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_lease": "5.000000", 2024-03-30T12:55:39.780 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-03-30T12:55:39.780 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-03-30T12:55:39.780 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_log_full_interval": "50", 2024-03-30T12:55:39.780 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_log_max": "10000", 2024-03-30T12:55:39.780 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_log_max_summary": "50", 2024-03-30T12:55:39.781 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_max_log_entries_per_event": "4096", 2024-03-30T12:55:39.781 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_max_log_epochs": "500", 2024-03-30T12:55:39.781 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_max_mdsmap_epochs": "500", 2024-03-30T12:55:39.781 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_max_mgrmap_epochs": "500", 2024-03-30T12:55:39.781 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_max_osd": "10000", 2024-03-30T12:55:39.781 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_max_pg_per_osd": "10000", 2024-03-30T12:55:39.781 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_max_pool_pg_num": "65536", 2024-03-30T12:55:39.781 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-03-30T12:55:39.781 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-03-30T12:55:39.781 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_mds_force_trim_to": "0", 2024-03-30T12:55:39.781 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_mds_skip_sanity": "false", 2024-03-30T12:55:39.781 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_memory_autotune": "true", 2024-03-30T12:55:39.781 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_memory_target": "2147483648", 2024-03-30T12:55:39.781 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_mgr_beacon_grace": "30", 2024-03-30T12:55:39.781 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-03-30T12:55:39.782 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_mgr_digest_period": "5", 2024-03-30T12:55:39.782 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_mgr_inactive_grace": "60", 2024-03-30T12:55:39.782 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_mgr_mkfs_grace": "240", 2024-03-30T12:55:39.782 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-03-30T12:55:39.782 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_min_osdmap_epochs": "500", 2024-03-30T12:55:39.782 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_op_complaint_time": "30", 2024-03-30T12:55:39.782 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_op_history_duration": "600", 2024-03-30T12:55:39.782 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_op_history_size": "20", 2024-03-30T12:55:39.782 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_op_history_slow_op_size": "20", 2024-03-30T12:55:39.782 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_op_history_slow_op_threshold": "10", 2024-03-30T12:55:39.782 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_op_log_threshold": "5", 2024-03-30T12:55:39.782 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-03-30T12:55:39.782 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-03-30T12:55:39.783 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-03-30T12:55:39.783 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_auto_mark_in": "false", 2024-03-30T12:55:39.783 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_auto_mark_new_in": "true", 2024-03-30T12:55:39.783 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-03-30T12:55:39.783 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-03-30T12:55:39.783 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_cache_size": "500", 2024-03-30T12:55:39.783 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_cache_size_min": "134217728", 2024-03-30T12:55:39.783 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_crush_smoke_test": "true", 2024-03-30T12:55:39.783 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_destroyed_out_interval": "600", 2024-03-30T12:55:39.783 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_down_out_interval": "600", 2024-03-30T12:55:39.783 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-03-30T12:55:39.783 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-03-30T12:55:39.783 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_force_trim_to": "0", 2024-03-30T12:55:39.783 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_full_ratio": "0.950000", 2024-03-30T12:55:39.784 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-03-30T12:55:39.784 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_laggy_halflife": "3600", 2024-03-30T12:55:39.784 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_laggy_max_interval": "300", 2024-03-30T12:55:39.784 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_laggy_weight": "0.300000", 2024-03-30T12:55:39.784 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-03-30T12:55:39.784 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_max_creating_pgs": "1024", 2024-03-30T12:55:39.784 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_max_initial_pgs": "1024", 2024-03-30T12:55:39.784 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_min_down_reporters": "2", 2024-03-30T12:55:39.784 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-03-30T12:55:39.784 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-03-30T12:55:39.784 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-03-30T12:55:39.784 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_prime_pg_temp": "true", 2024-03-30T12:55:39.784 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-03-30T12:55:39.784 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-03-30T12:55:39.784 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_report_timeout": "900", 2024-03-30T12:55:39.784 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-03-30T12:55:39.785 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-03-30T12:55:39.785 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_warn_num_repaired": "10", 2024-03-30T12:55:39.785 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osd_warn_op_age": "32.000000", 2024-03-30T12:55:39.785 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-03-30T12:55:39.785 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osdmap_full_prune_interval": "10", 2024-03-30T12:55:39.785 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osdmap_full_prune_min": "10000", 2024-03-30T12:55:39.785 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-03-30T12:55:39.785 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-03-30T12:55:39.785 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_pg_stuck_threshold": "60", 2024-03-30T12:55:39.785 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-03-30T12:55:39.785 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_pg_warn_min_objects": "10000", 2024-03-30T12:55:39.785 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_pg_warn_min_per_osd": "0", 2024-03-30T12:55:39.785 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-03-30T12:55:39.785 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_pool_quota_crit_threshold": "0", 2024-03-30T12:55:39.786 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_pool_quota_warn_threshold": "0", 2024-03-30T12:55:39.786 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_probe_timeout": "2.000000", 2024-03-30T12:55:39.786 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_reweight_max_change": "0.050000", 2024-03-30T12:55:39.786 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_reweight_max_osds": "4", 2024-03-30T12:55:39.786 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-03-30T12:55:39.786 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-03-30T12:55:39.786 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-03-30T12:55:39.786 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-03-30T12:55:39.786 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-03-30T12:55:39.786 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_scrub_interval": "300", 2024-03-30T12:55:39.786 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_scrub_max_keys": "100", 2024-03-30T12:55:39.786 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_scrub_timeout": "300", 2024-03-30T12:55:39.786 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_session_timeout": "300", 2024-03-30T12:55:39.786 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_smart_report_timeout": "5", 2024-03-30T12:55:39.787 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_stat_smooth_intervals": "6", 2024-03-30T12:55:39.787 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-03-30T12:55:39.787 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_stretch_pool_min_size": "2", 2024-03-30T12:55:39.787 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_stretch_pool_size": "4", 2024-03-30T12:55:39.787 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-03-30T12:55:39.787 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_subscribe_interval": "86400.000000", 2024-03-30T12:55:39.787 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_sync_debug": "false", 2024-03-30T12:55:39.787 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_sync_max_payload_keys": "2000", 2024-03-30T12:55:39.787 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_sync_max_payload_size": "1048576", 2024-03-30T12:55:39.787 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_sync_provider_kill_at": "0", 2024-03-30T12:55:39.787 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_sync_requester_kill_at": "0", 2024-03-30T12:55:39.787 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_sync_timeout": "60.000000", 2024-03-30T12:55:39.787 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_target_pg_per_osd": "100", 2024-03-30T12:55:39.788 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_tick_interval": "5", 2024-03-30T12:55:39.788 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_timecheck_interval": "300.000000", 2024-03-30T12:55:39.788 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-03-30T12:55:39.788 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_use_min_delay_socket": "false", 2024-03-30T12:55:39.788 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_older_version_delay": "604800", 2024-03-30T12:55:39.788 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-03-30T12:55:39.788 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-03-30T12:55:39.788 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-03-30T12:55:39.788 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_filestore_osds": "false", 2024-03-30T12:55:39.788 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-03-30T12:55:39.788 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-03-30T12:55:39.788 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-03-30T12:55:39.788 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_misplaced": "false", 2024-03-30T12:55:39.788 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-03-30T12:55:39.788 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_older_version": "true", 2024-03-30T12:55:39.789 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-03-30T12:55:39.789 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_pool_no_app": "true", 2024-03-30T12:55:39.789 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-03-30T12:55:39.789 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-03-30T12:55:39.789 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-03-30T12:55:39.789 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-03-30T12:55:39.789 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_on_too_few_osds": "false", 2024-03-30T12:55:39.789 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-03-30T12:55:39.789 INFO:teuthology.orchestra.run.smithi067.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-03-30T12:55:39.789 INFO:teuthology.orchestra.run.smithi067.stdout: "monmap": "", 2024-03-30T12:55:39.789 INFO:teuthology.orchestra.run.smithi067.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-30T12:55:39.790 INFO:teuthology.orchestra.run.smithi067.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-03-30T12:55:39.790 INFO:teuthology.orchestra.run.smithi067.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-03-30T12:55:39.790 INFO:teuthology.orchestra.run.smithi067.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-30T12:55:39.790 INFO:teuthology.orchestra.run.smithi067.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-03-30T12:55:39.790 INFO:teuthology.orchestra.run.smithi067.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-03-30T12:55:39.790 INFO:teuthology.orchestra.run.smithi067.stdout: "motr_tracing_enabled": "false", 2024-03-30T12:55:39.790 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_op_threads": "3", 2024-03-30T12:55:39.790 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_buffer_size": "131072", 2024-03-30T12:55:39.790 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_cm": "false", 2024-03-30T12:55:39.790 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_device_name": "", 2024-03-30T12:55:39.790 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_dscp": "96", 2024-03-30T12:55:39.790 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-03-30T12:55:39.790 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_gid_idx": "0", 2024-03-30T12:55:39.790 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_local_gid": "", 2024-03-30T12:55:39.791 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_polling_us": "1000", 2024-03-30T12:55:39.791 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_port_num": "1", 2024-03-30T12:55:39.791 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-03-30T12:55:39.791 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-03-30T12:55:39.791 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_roce_ver": "1", 2024-03-30T12:55:39.791 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_send_buffers": "1024", 2024-03-30T12:55:39.791 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_sl": "3", 2024-03-30T12:55:39.791 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_support_srq": "true", 2024-03-30T12:55:39.791 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_rdma_type": "ib", 2024-03-30T12:55:39.791 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_async_reap_threshold": "5", 2024-03-30T12:55:39.791 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_before_connect": "false", 2024-03-30T12:55:39.791 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_ipv4": "true", 2024-03-30T12:55:39.791 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_ipv6": "false", 2024-03-30T12:55:39.791 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_msgr1": "true", 2024-03-30T12:55:39.791 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_msgr2": "true", 2024-03-30T12:55:39.791 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_port_max": "7568", 2024-03-30T12:55:39.792 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_port_min": "6800", 2024-03-30T12:55:39.792 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_prefer_ipv4": "false", 2024-03-30T12:55:39.792 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_retry_count": "3", 2024-03-30T12:55:39.792 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_bind_retry_delay": "5", 2024-03-30T12:55:39.792 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_blackhole_client": "false", 2024-03-30T12:55:39.792 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_blackhole_mds": "false", 2024-03-30T12:55:39.792 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_blackhole_mgr": "false", 2024-03-30T12:55:39.792 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_blackhole_mon": "false", 2024-03-30T12:55:39.792 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_blackhole_osd": "false", 2024-03-30T12:55:39.792 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_client_mode": "crc secure", 2024-03-30T12:55:39.792 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_cluster_mode": "crc secure", 2024-03-30T12:55:39.792 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_cluster_type": "", 2024-03-30T12:55:39.792 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_compress_secure": "false", 2024-03-30T12:55:39.792 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_connection_idle_timeout": "900", 2024-03-30T12:55:39.793 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_connection_ready_timeout": "10", 2024-03-30T12:55:39.793 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_crc_data": "true", 2024-03-30T12:55:39.793 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_crc_header": "true", 2024-03-30T12:55:39.793 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_die_on_bad_msg": "false", 2024-03-30T12:55:39.793 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_die_on_bug": "true", 2024-03-30T12:55:39.793 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_die_on_old_message": "true", 2024-03-30T12:55:39.793 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_die_on_skipped_message": "false", 2024-03-30T12:55:39.793 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_die_on_unhandled_msg": "false", 2024-03-30T12:55:39.793 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-03-30T12:55:39.793 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_coremask": "0xF", 2024-03-30T12:55:39.793 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-03-30T12:55:39.793 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_devs_allowlist": "", 2024-03-30T12:55:39.793 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_enable_tso": "true", 2024-03-30T12:55:39.793 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-03-30T12:55:39.793 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-03-30T12:55:39.794 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_hugepages": "", 2024-03-30T12:55:39.794 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_hw_flow_control": "true", 2024-03-30T12:55:39.794 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-03-30T12:55:39.794 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_lro": "true", 2024-03-30T12:55:39.794 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_memory_channel": "4", 2024-03-30T12:55:39.794 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-03-30T12:55:39.794 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_pmd": "", 2024-03-30T12:55:39.794 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_port_id": "0", 2024-03-30T12:55:39.794 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-03-30T12:55:39.794 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dump_corrupt_message_level": "1", 2024-03-30T12:55:39.794 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_dump_on_send": "false", 2024-03-30T12:55:39.794 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_initial_backoff": "0.200000", 2024-03-30T12:55:39.794 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_inject_delay_max": "1.000000", 2024-03-30T12:55:39.794 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_inject_delay_probability": "0.005000", 2024-03-30T12:55:39.794 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_inject_delay_type": "osd", 2024-03-30T12:55:39.795 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_inject_internal_delays": "0.002000", 2024-03-30T12:55:39.795 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_inject_network_congestion": "0", 2024-03-30T12:55:39.795 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_inject_socket_failures": "2500", 2024-03-30T12:55:39.795 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_learn_addr_from_peer": "true", 2024-03-30T12:55:39.795 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_max_accept_failures": "4", 2024-03-30T12:55:39.795 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_max_backoff": "15.000000", 2024-03-30T12:55:39.795 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_mon_client_mode": "secure crc", 2024-03-30T12:55:39.795 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_mon_cluster_mode": "secure crc", 2024-03-30T12:55:39.795 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_mon_service_mode": "secure crc", 2024-03-30T12:55:39.795 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_osd_compress_min_size": "1024", 2024-03-30T12:55:39.795 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_osd_compress_mode": "none", 2024-03-30T12:55:39.795 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_osd_compression_algorithm": "snappy", 2024-03-30T12:55:39.795 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-03-30T12:55:39.795 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_pq_min_cost": "65536", 2024-03-30T12:55:39.795 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_public_type": "", 2024-03-30T12:55:39.796 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_service_mode": "crc secure", 2024-03-30T12:55:39.796 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_tcp_listen_backlog": "512", 2024-03-30T12:55:39.796 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_tcp_nodelay": "true", 2024-03-30T12:55:39.796 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-03-30T12:55:39.796 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_tcp_rcvbuf": "0", 2024-03-30T12:55:39.796 INFO:teuthology.orchestra.run.smithi067.stdout: "ms_type": "async+posix", 2024-03-30T12:55:39.796 INFO:teuthology.orchestra.run.smithi067.stdout: "no_config_file": "false", 2024-03-30T12:55:39.796 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter_completion_locks_per_session": "32", 2024-03-30T12:55:39.796 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter_debug_inject_relock_delay": "false", 2024-03-30T12:55:39.796 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter_inflight_op_bytes": "104857600", 2024-03-30T12:55:39.796 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter_inflight_ops": "1024", 2024-03-30T12:55:39.796 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter_inject_no_watch_ping": "false", 2024-03-30T12:55:39.796 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-03-30T12:55:39.796 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter_tick_interval": "5.000000", 2024-03-30T12:55:39.796 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter_timeout": "10.000000", 2024-03-30T12:55:39.796 INFO:teuthology.orchestra.run.smithi067.stdout: "objectstore_blackhole": "false", 2024-03-30T12:55:39.797 INFO:teuthology.orchestra.run.smithi067.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-03-30T12:55:39.797 INFO:teuthology.orchestra.run.smithi067.stdout: "openssl_engine_opts": "", 2024-03-30T12:55:39.797 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_agent_delay_time": "5.000000", 2024-03-30T12:55:39.797 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_agent_hist_halflife": "1000", 2024-03-30T12:55:39.797 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_agent_max_low_ops": "2", 2024-03-30T12:55:39.797 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_agent_max_ops": "4", 2024-03-30T12:55:39.797 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-03-30T12:55:39.797 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_agent_quantize_effort": "0.100000", 2024-03-30T12:55:39.797 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_agent_slop": "0.020000", 2024-03-30T12:55:39.797 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-03-30T12:55:39.797 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_allow_recovery_below_min_size": "true", 2024-03-30T12:55:39.797 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_asio_thread_count": "2", 2024-03-30T12:55:39.797 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_async_recovery_min_cost": "100", 2024-03-30T12:55:39.797 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_auto_mark_unfound_lost": "false", 2024-03-30T12:55:39.797 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_backfill_retry_interval": "30.000000", 2024-03-30T12:55:39.798 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_backfill_scan_max": "512", 2024-03-30T12:55:39.798 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_backfill_scan_min": "64", 2024-03-30T12:55:39.798 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_backoff_on_degraded": "false", 2024-03-30T12:55:39.798 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_backoff_on_peering": "false", 2024-03-30T12:55:39.798 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_backoff_on_unfound": "true", 2024-03-30T12:55:39.798 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_beacon_report_interval": "300", 2024-03-30T12:55:39.798 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_bench_duration": "30", 2024-03-30T12:55:39.798 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-03-30T12:55:39.798 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_bench_max_block_size": "67108864", 2024-03-30T12:55:39.798 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_bench_small_size_max_iops": "100", 2024-03-30T12:55:39.798 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_blkin_trace_all": "false", 2024-03-30T12:55:39.798 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_blocked_scrub_grace_period": "120", 2024-03-30T12:55:39.798 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-03-30T12:55:39.798 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-03-30T12:55:39.799 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-03-30T12:55:39.799 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_check_for_log_corruption": "false", 2024-03-30T12:55:39.799 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-03-30T12:55:39.799 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-30T12:55:39.799 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2024-03-30T12:55:39.799 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-30T12:55:39.799 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_class_update_on_start": "true", 2024-03-30T12:55:39.799 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_client_message_cap": "256", 2024-03-30T12:55:39.799 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_client_message_size_cap": "524288000", 2024-03-30T12:55:39.799 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_client_op_priority": "63", 2024-03-30T12:55:39.799 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_client_watch_timeout": "30", 2024-03-30T12:55:39.799 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_command_max_records": "256", 2024-03-30T12:55:39.799 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_compact_on_start": "false", 2024-03-30T12:55:39.799 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-03-30T12:55:39.799 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_crush_chooseleaf_type": "0", 2024-03-30T12:55:39.800 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_crush_initial_weight": "-1.000000", 2024-03-30T12:55:39.800 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_crush_update_on_start": "true", 2024-03-30T12:55:39.800 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_crush_update_weight_set": "true", 2024-03-30T12:55:39.800 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-03-30T12:55:39.800 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-03-30T12:55:39.800 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-03-30T12:55:39.800 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_drop_ping_duration": "0", 2024-03-30T12:55:39.800 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-03-30T12:55:39.800 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_feed_pullee": "-1", 2024-03-30T12:55:39.800 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-03-30T12:55:39.800 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-03-30T12:55:39.800 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-03-30T12:55:39.800 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_misdirected_ops": "false", 2024-03-30T12:55:39.800 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_no_acting_change": "false", 2024-03-30T12:55:39.800 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_no_purge_strays": "false", 2024-03-30T12:55:39.800 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_op_order": "false", 2024-03-30T12:55:39.801 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_pg_log_writeout": "false", 2024-03-30T12:55:39.801 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_pretend_recovery_active": "false", 2024-03-30T12:55:39.801 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-03-30T12:55:39.801 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-03-30T12:55:39.801 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_shutdown": "false", 2024-03-30T12:55:39.801 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-03-30T12:55:39.801 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-03-30T12:55:39.801 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_trim_objects": "false", 2024-03-30T12:55:39.801 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_verify_cached_snaps": "false", 2024-03-30T12:55:39.801 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_verify_missing_on_start": "false", 2024-03-30T12:55:39.801 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_verify_snaps": "false", 2024-03-30T12:55:39.801 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-03-30T12:55:39.801 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-03-30T12:55:39.801 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_deep_scrub_keys": "1024", 2024-03-30T12:55:39.802 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-03-30T12:55:39.802 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-03-30T12:55:39.802 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-03-30T12:55:39.802 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_deep_scrub_stride": "524288", 2024-03-30T12:55:39.802 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-03-30T12:55:39.802 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_default_data_pool_replay_window": "5", 2024-03-30T12:55:39.802 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_default_notify_timeout": "30", 2024-03-30T12:55:39.802 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_delete_sleep": "0.000000", 2024-03-30T12:55:39.802 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-03-30T12:55:39.802 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-03-30T12:55:39.802 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-03-30T12:55:39.802 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_discard_disconnected_ops": "true", 2024-03-30T12:55:39.802 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_enable_op_tracker": "true", 2024-03-30T12:55:39.802 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-03-30T12:55:39.802 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_extblkdev_plugins": "vdo", 2024-03-30T12:55:39.803 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-03-30T12:55:39.803 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-03-30T12:55:39.803 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_fast_info": "true", 2024-03-30T12:55:39.803 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_fast_shutdown": "true", 2024-03-30T12:55:39.803 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-03-30T12:55:39.803 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_fast_shutdown_timeout": "15", 2024-03-30T12:55:39.803 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-03-30T12:55:39.803 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-03-30T12:55:39.803 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-03-30T12:55:39.803 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_function_tracing": "false", 2024-03-30T12:55:39.803 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_heartbeat_grace": "20", 2024-03-30T12:55:39.803 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_heartbeat_interval": "6", 2024-03-30T12:55:39.803 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-03-30T12:55:39.803 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_heartbeat_min_peers": "10", 2024-03-30T12:55:39.804 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_heartbeat_min_size": "2000", 2024-03-30T12:55:39.804 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_heartbeat_stale": "600", 2024-03-30T12:55:39.804 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-03-30T12:55:39.804 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_hit_set_max_size": "100000", 2024-03-30T12:55:39.804 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_hit_set_min_size": "1000", 2024-03-30T12:55:39.804 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-03-30T12:55:39.804 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-03-30T12:55:39.804 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-03-30T12:55:39.804 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-03-30T12:55:39.804 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-03-30T12:55:39.804 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_journal_flush_on_shutdown": "true", 2024-03-30T12:55:39.804 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_journal_size": "5120", 2024-03-30T12:55:39.804 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_kill_backfill_at": "0", 2024-03-30T12:55:39.804 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_loop_before_reset_tphandle": "64", 2024-03-30T12:55:39.804 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_map_cache_size": "50", 2024-03-30T12:55:39.804 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_map_dedup": "true", 2024-03-30T12:55:39.805 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_map_message_max": "40", 2024-03-30T12:55:39.805 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_map_message_max_bytes": "10485760", 2024-03-30T12:55:39.805 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_map_share_max_epochs": "40", 2024-03-30T12:55:39.805 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_attr_name_len": "100", 2024-03-30T12:55:39.805 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_attr_size": "0", 2024-03-30T12:55:39.805 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_backfills": "1", 2024-03-30T12:55:39.805 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_markdown_count": "5", 2024-03-30T12:55:39.805 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_markdown_period": "600", 2024-03-30T12:55:39.805 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_object_name_len": "2048", 2024-03-30T12:55:39.805 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_object_namespace_len": "256", 2024-03-30T12:55:39.805 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_object_size": "134217728", 2024-03-30T12:55:39.805 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-03-30T12:55:39.805 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_omap_entries_per_request": "1024", 2024-03-30T12:55:39.805 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_pg_blocked_by": "16", 2024-03-30T12:55:39.805 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_pg_log_entries": "10000", 2024-03-30T12:55:39.806 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-03-30T12:55:39.806 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_pgls": "1024", 2024-03-30T12:55:39.806 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_push_cost": "8388608", 2024-03-30T12:55:39.806 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_push_objects": "10", 2024-03-30T12:55:39.806 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_scrubs": "3", 2024-03-30T12:55:39.806 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-03-30T12:55:39.806 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_trimming_pgs": "2", 2024-03-30T12:55:39.806 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_write_op_reply_len": "64", 2024-03-30T12:55:39.806 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_max_write_size": "90", 2024-03-30T12:55:39.806 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-03-30T12:55:39.806 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-03-30T12:55:39.806 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-03-30T12:55:39.806 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-03-30T12:55:39.806 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-03-30T12:55:39.807 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-03-30T12:55:39.807 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-03-30T12:55:39.807 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_override_recovery_settings": "false", 2024-03-30T12:55:39.807 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_profile": "balanced", 2024-03-30T12:55:39.807 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-03-30T12:55:39.807 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-03-30T12:55:39.807 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-03-30T12:55:39.807 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-03-30T12:55:39.807 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-03-30T12:55:39.807 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-03-30T12:55:39.807 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-03-30T12:55:39.807 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-03-30T12:55:39.807 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-03-30T12:55:39.807 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-03-30T12:55:39.807 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mclock_skip_benchmark": "false", 2024-03-30T12:55:39.808 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_memory_base": "805306368", 2024-03-30T12:55:39.808 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_memory_cache_min": "134217728", 2024-03-30T12:55:39.808 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-03-30T12:55:39.808 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-03-30T12:55:39.808 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_memory_target": "4294967296", 2024-03-30T12:55:39.808 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_memory_target_autotune": "false", 2024-03-30T12:55:39.808 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-03-30T12:55:39.808 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_min_pg_log_entries": "250", 2024-03-30T12:55:39.808 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_min_recovery_priority": "0", 2024-03-30T12:55:39.808 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mon_heartbeat_interval": "30", 2024-03-30T12:55:39.808 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-03-30T12:55:39.808 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mon_report_interval": "5", 2024-03-30T12:55:39.808 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mon_report_max_in_flight": "2", 2024-03-30T12:55:39.808 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-03-30T12:55:39.808 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_num_cache_shards": "32", 2024-03-30T12:55:39.809 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_num_op_tracker_shard": "32", 2024-03-30T12:55:39.809 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_numa_auto_affinity": "true", 2024-03-30T12:55:39.809 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_numa_node": "-1", 2024-03-30T12:55:39.809 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_numa_prefer_iface": "true", 2024-03-30T12:55:39.809 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-03-30T12:55:39.809 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_objecter_finishers": "1", 2024-03-30T12:55:39.809 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_objectstore": "bluestore", 2024-03-30T12:55:39.809 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_objectstore_fuse": "false", 2024-03-30T12:55:39.809 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_objectstore_tracing": "false", 2024-03-30T12:55:39.809 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_complaint_time": "30.000000", 2024-03-30T12:55:39.809 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_history_duration": "600", 2024-03-30T12:55:39.809 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_history_size": "20", 2024-03-30T12:55:39.809 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_history_slow_op_size": "20", 2024-03-30T12:55:39.809 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-03-30T12:55:39.809 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_log_threshold": "5", 2024-03-30T12:55:39.810 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_num_shards": "0", 2024-03-30T12:55:39.810 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_num_shards_hdd": "5", 2024-03-30T12:55:39.810 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_num_shards_ssd": "8", 2024-03-30T12:55:39.810 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_num_threads_per_shard": "0", 2024-03-30T12:55:39.810 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-03-30T12:55:39.810 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-03-30T12:55:39.810 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-03-30T12:55:39.810 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_pq_min_cost": "65536", 2024-03-30T12:55:39.810 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_queue": "mclock_scheduler", 2024-03-30T12:55:39.810 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_queue_cut_off": "high", 2024-03-30T12:55:39.810 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_thread_suicide_timeout": "150", 2024-03-30T12:55:39.810 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_op_thread_timeout": "15", 2024-03-30T12:55:39.810 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_open_classes_on_start": "true", 2024-03-30T12:55:39.810 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_os_flags": "0", 2024-03-30T12:55:39.810 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_peering_op_priority": "255", 2024-03-30T12:55:39.811 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_delete_cost": "1048576", 2024-03-30T12:55:39.811 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_delete_priority": "5", 2024-03-30T12:55:39.811 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-03-30T12:55:39.811 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-03-30T12:55:39.811 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_log_dups_tracked": "3000", 2024-03-30T12:55:39.811 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_log_trim_max": "10000", 2024-03-30T12:55:39.811 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_log_trim_min": "100", 2024-03-30T12:55:39.811 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-03-30T12:55:39.811 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_object_context_cache_count": "64", 2024-03-30T12:55:39.811 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-03-30T12:55:39.811 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-03-30T12:55:39.811 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-03-30T12:55:39.811 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-03-30T12:55:39.811 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-03-30T12:55:39.811 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-03-30T12:55:39.812 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-03-30T12:55:39.812 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-03-30T12:55:39.812 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_crimson": "false", 2024-03-30T12:55:39.812 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_crush_rule": "-1", 2024-03-30T12:55:39.812 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_ec_fast_read": "false", 2024-03-30T12:55:39.812 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-03-30T12:55:39.812 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_flag_bulk": "false", 2024-03-30T12:55:39.812 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-03-30T12:55:39.812 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_flag_nodelete": "false", 2024-03-30T12:55:39.812 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-03-30T12:55:39.812 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-03-30T12:55:39.812 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_flags": "0", 2024-03-30T12:55:39.812 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-03-30T12:55:39.812 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_min_size": "0", 2024-03-30T12:55:39.812 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-03-30T12:55:39.813 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_pg_num": "32", 2024-03-30T12:55:39.813 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_pgp_num": "0", 2024-03-30T12:55:39.813 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-03-30T12:55:39.813 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_read_ratio": "70", 2024-03-30T12:55:39.813 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_size": "2", 2024-03-30T12:55:39.813 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_default_type": "replicated", 2024-03-30T12:55:39.813 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-03-30T12:55:39.813 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pool_use_gmt_hitset": "true", 2024-03-30T12:55:39.813 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_push_per_object_cost": "1000", 2024-03-30T12:55:39.813 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_read_ec_check_for_errors": "false", 2024-03-30T12:55:39.813 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recover_clone_overlap": "true", 2024-03-30T12:55:39.813 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recover_clone_overlap_limit": "10", 2024-03-30T12:55:39.813 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_cost": "20971520", 2024-03-30T12:55:39.813 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_delay_start": "0.000000", 2024-03-30T12:55:39.814 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_max_active": "0", 2024-03-30T12:55:39.814 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_max_active_hdd": "3", 2024-03-30T12:55:39.814 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_max_active_ssd": "10", 2024-03-30T12:55:39.814 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_max_chunk": "8388608", 2024-03-30T12:55:39.814 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-03-30T12:55:39.814 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_max_single_start": "1", 2024-03-30T12:55:39.814 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_op_priority": "3", 2024-03-30T12:55:39.814 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_op_warn_multiple": "16", 2024-03-30T12:55:39.814 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_priority": "5", 2024-03-30T12:55:39.814 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_retry_interval": "30.000000", 2024-03-30T12:55:39.814 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_sleep": "0.000000", 2024-03-30T12:55:39.814 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-03-30T12:55:39.814 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-03-30T12:55:39.814 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-03-30T12:55:39.814 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_repair_during_recovery": "false", 2024-03-30T12:55:39.815 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_requested_scrub_priority": "120", 2024-03-30T12:55:39.815 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-03-30T12:55:39.815 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_rollback_to_cluster_snap": "", 2024-03-30T12:55:39.815 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_auto_repair": "false", 2024-03-30T12:55:39.815 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-03-30T12:55:39.815 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-03-30T12:55:39.815 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_begin_hour": "0", 2024-03-30T12:55:39.815 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_begin_week_day": "0", 2024-03-30T12:55:39.815 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_chunk_max": "25", 2024-03-30T12:55:39.815 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_chunk_min": "5", 2024-03-30T12:55:39.815 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_cost": "52428800", 2024-03-30T12:55:39.815 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-03-30T12:55:39.815 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_during_recovery": "false", 2024-03-30T12:55:39.815 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_end_hour": "0", 2024-03-30T12:55:39.815 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_end_week_day": "0", 2024-03-30T12:55:39.816 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_event_cost": "4096", 2024-03-30T12:55:39.816 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-03-30T12:55:39.816 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-03-30T12:55:39.816 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_invalid_stats": "true", 2024-03-30T12:55:39.816 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_load_threshold": "0.500000", 2024-03-30T12:55:39.816 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_max_interval": "604800.000000", 2024-03-30T12:55:39.816 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_max_preemptions": "5", 2024-03-30T12:55:39.816 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_min_interval": "86400.000000", 2024-03-30T12:55:39.816 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_priority": "5", 2024-03-30T12:55:39.816 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_reservation_timeout": "300000", 2024-03-30T12:55:39.816 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_sleep": "0.000000", 2024-03-30T12:55:39.816 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-03-30T12:55:39.816 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-03-30T12:55:39.816 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-03-30T12:55:39.816 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_shutdown_pgref_assert": "false", 2024-03-30T12:55:39.817 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-03-30T12:55:39.817 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_skip_data_digest": "false", 2024-03-30T12:55:39.817 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_smart_report_timeout": "5", 2024-03-30T12:55:39.817 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_snap_trim_cost": "1048576", 2024-03-30T12:55:39.817 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_snap_trim_priority": "5", 2024-03-30T12:55:39.817 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_snap_trim_sleep": "0.000000", 2024-03-30T12:55:39.817 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-03-30T12:55:39.817 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-03-30T12:55:39.817 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-03-30T12:55:39.817 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-03-30T12:55:39.817 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_stats_update_period_scrubbing": "15", 2024-03-30T12:55:39.817 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-03-30T12:55:39.817 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_target_transaction_size": "30", 2024-03-30T12:55:39.817 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-03-30T12:55:39.817 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-03-30T12:55:39.818 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-03-30T12:55:39.818 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-03-30T12:55:39.818 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-03-30T12:55:39.818 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-03-30T12:55:39.818 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-03-30T12:55:39.818 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_default_cache_mode": "writeback", 2024-03-30T12:55:39.818 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-03-30T12:55:39.818 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-03-30T12:55:39.818 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_tracing": "false", 2024-03-30T12:55:39.818 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_use_stale_snap": "false", 2024-03-30T12:55:39.818 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-03-30T12:55:39.818 INFO:teuthology.orchestra.run.smithi067.stdout: "osdc_blkin_trace_all": "false", 2024-03-30T12:55:39.818 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_kill_at": "0", 2024-03-30T12:55:39.818 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_max_join_drift": "10", 2024-03-30T12:55:39.819 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_min": "500", 2024-03-30T12:55:39.819 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_min_wait": "0.050000", 2024-03-30T12:55:39.819 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_propose_interval": "1.000000", 2024-03-30T12:55:39.819 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_service_trim_max": "500", 2024-03-30T12:55:39.819 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_service_trim_max_multiplier": "20", 2024-03-30T12:55:39.819 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_service_trim_min": "250", 2024-03-30T12:55:39.819 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_stash_full_interval": "25", 2024-03-30T12:55:39.819 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_trim_max": "500", 2024-03-30T12:55:39.819 INFO:teuthology.orchestra.run.smithi067.stdout: "paxos_trim_min": "250", 2024-03-30T12:55:39.819 INFO:teuthology.orchestra.run.smithi067.stdout: "perf": "true", 2024-03-30T12:55:39.819 INFO:teuthology.orchestra.run.smithi067.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-03-30T12:55:39.819 INFO:teuthology.orchestra.run.smithi067.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-03-30T12:55:39.819 INFO:teuthology.orchestra.run.smithi067.stdout: "plugin_dir": "/usr/lib/ceph", 2024-03-30T12:55:39.819 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addr": "-", 2024-03-30T12:55:39.819 INFO:teuthology.orchestra.run.smithi067.stdout: "public_addrv": "", 2024-03-30T12:55:39.820 INFO:teuthology.orchestra.run.smithi067.stdout: "public_bind_addr": "-", 2024-03-30T12:55:39.820 INFO:teuthology.orchestra.run.smithi067.stdout: "public_network": "", 2024-03-30T12:55:39.820 INFO:teuthology.orchestra.run.smithi067.stdout: "public_network_interface": "", 2024-03-30T12:55:39.820 INFO:teuthology.orchestra.run.smithi067.stdout: "qat_compressor_enabled": "false", 2024-03-30T12:55:39.820 INFO:teuthology.orchestra.run.smithi067.stdout: "qat_compressor_session_max_number": "256", 2024-03-30T12:55:39.820 INFO:teuthology.orchestra.run.smithi067.stdout: "rados_mon_op_timeout": "0", 2024-03-30T12:55:39.820 INFO:teuthology.orchestra.run.smithi067.stdout: "rados_osd_op_timeout": "0", 2024-03-30T12:55:39.820 INFO:teuthology.orchestra.run.smithi067.stdout: "rados_tracing": "false", 2024-03-30T12:55:39.820 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_atime_update_interval": "60", 2024-03-30T12:55:39.820 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-03-30T12:55:39.820 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_balance_parent_reads": "false", 2024-03-30T12:55:39.820 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_balance_snap_reads": "false", 2024-03-30T12:55:39.820 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_blkin_trace_all": "false", 2024-03-30T12:55:39.820 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_blocklist_expire_seconds": "0", 2024-03-30T12:55:39.820 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_blocklist_on_break_lock": "true", 2024-03-30T12:55:39.821 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_cache": "true", 2024-03-30T12:55:39.821 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_cache_block_writes_upfront": "false", 2024-03-30T12:55:39.821 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_cache_max_dirty": "25165824", 2024-03-30T12:55:39.821 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-03-30T12:55:39.821 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_cache_max_dirty_object": "0", 2024-03-30T12:55:39.821 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_cache_policy": "writearound", 2024-03-30T12:55:39.821 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_cache_size": "33554432", 2024-03-30T12:55:39.821 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_cache_target_dirty": "16777216", 2024-03-30T12:55:39.821 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-03-30T12:55:39.821 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_clone_copy_on_read": "false", 2024-03-30T12:55:39.821 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_compression_hint": "none", 2024-03-30T12:55:39.821 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_concurrent_management_ops": "10", 2024-03-30T12:55:39.821 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-03-30T12:55:39.821 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_clone_format": "auto", 2024-03-30T12:55:39.821 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_data_pool": "", 2024-03-30T12:55:39.822 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_features": "61", 2024-03-30T12:55:39.822 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_format": "2", 2024-03-30T12:55:39.822 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_map_options": "", 2024-03-30T12:55:39.822 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_order": "22", 2024-03-30T12:55:39.822 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_pool": "rbd", 2024-03-30T12:55:39.822 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-03-30T12:55:39.822 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_stripe_count": "0", 2024-03-30T12:55:39.822 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_default_stripe_unit": "0", 2024-03-30T12:55:39.822 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_disable_zero_copy_writes": "true", 2024-03-30T12:55:39.822 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_discard_granularity_bytes": "65536", 2024-03-30T12:55:39.822 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-03-30T12:55:39.822 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_enable_alloc_hint": "true", 2024-03-30T12:55:39.822 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-03-30T12:55:39.822 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_io_scheduler": "simple", 2024-03-30T12:55:39.822 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-03-30T12:55:39.823 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_commit_age": "5.000000", 2024-03-30T12:55:39.823 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-03-30T12:55:39.823 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-03-30T12:55:39.823 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-03-30T12:55:39.823 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-03-30T12:55:39.823 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_object_flush_interval": "0", 2024-03-30T12:55:39.823 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-03-30T12:55:39.823 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-03-30T12:55:39.823 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_order": "24", 2024-03-30T12:55:39.823 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_pool": "", 2024-03-30T12:55:39.823 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_journal_splay_width": "4", 2024-03-30T12:55:39.823 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_localize_parent_reads": "false", 2024-03-30T12:55:39.823 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_localize_snap_reads": "false", 2024-03-30T12:55:39.823 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-03-30T12:55:39.823 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-03-30T12:55:39.824 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-03-30T12:55:39.824 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-03-30T12:55:39.824 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-03-30T12:55:39.824 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-03-30T12:55:39.824 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_image_policy_type": "simple", 2024-03-30T12:55:39.824 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-03-30T12:55:39.824 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-03-30T12:55:39.824 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-03-30T12:55:39.824 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-03-30T12:55:39.824 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-03-30T12:55:39.824 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-03-30T12:55:39.824 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-03-30T12:55:39.824 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_memory_autotune": "true", 2024-03-30T12:55:39.824 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_memory_base": "805306368", 2024-03-30T12:55:39.824 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-03-30T12:55:39.824 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-03-30T12:55:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-03-30T12:55:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-03-30T12:55:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_memory_target": "4294967296", 2024-03-30T12:55:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-03-30T12:55:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-03-30T12:55:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-03-30T12:55:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirroring_delete_delay": "0", 2024-03-30T12:55:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-03-30T12:55:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirroring_replay_delay": "0", 2024-03-30T12:55:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-03-30T12:55:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-03-30T12:55:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_move_to_trash_on_remove": "false", 2024-03-30T12:55:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-03-30T12:55:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_mtime_update_interval": "60", 2024-03-30T12:55:39.825 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_non_blocking_aio": "true", 2024-03-30T12:55:39.826 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_op_thread_timeout": "60", 2024-03-30T12:55:39.826 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_op_threads": "1", 2024-03-30T12:55:39.826 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_parent_cache_enabled": "false", 2024-03-30T12:55:39.826 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_persistent_cache_mode": "disabled", 2024-03-30T12:55:39.826 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_persistent_cache_path": "/tmp", 2024-03-30T12:55:39.826 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_persistent_cache_size": "1073741824", 2024-03-30T12:55:39.826 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_plugins": "", 2024-03-30T12:55:39.826 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_bps_burst": "0", 2024-03-30T12:55:39.826 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-03-30T12:55:39.826 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_bps_limit": "0", 2024-03-30T12:55:39.826 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_exclude_ops": "0", 2024-03-30T12:55:39.826 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_iops_burst": "0", 2024-03-30T12:55:39.826 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-03-30T12:55:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_iops_limit": "0", 2024-03-30T12:55:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_read_bps_burst": "0", 2024-03-30T12:55:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-03-30T12:55:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_read_bps_limit": "0", 2024-03-30T12:55:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_read_iops_burst": "0", 2024-03-30T12:55:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-03-30T12:55:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_read_iops_limit": "0", 2024-03-30T12:55:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_schedule_tick_min": "50", 2024-03-30T12:55:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_write_bps_burst": "0", 2024-03-30T12:55:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-03-30T12:55:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_write_bps_limit": "0", 2024-03-30T12:55:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_write_iops_burst": "0", 2024-03-30T12:55:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-03-30T12:55:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_qos_write_iops_limit": "0", 2024-03-30T12:55:39.827 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_quiesce_notification_attempts": "10", 2024-03-30T12:55:39.828 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_read_from_replica_policy": "default", 2024-03-30T12:55:39.828 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-03-30T12:55:39.828 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_readahead_max_bytes": "524288", 2024-03-30T12:55:39.828 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_readahead_trigger_requests": "10", 2024-03-30T12:55:39.828 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_request_timed_out_seconds": "30", 2024-03-30T12:55:39.828 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_skip_partial_discard": "true", 2024-03-30T12:55:39.828 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-03-30T12:55:39.828 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_tracing": "false", 2024-03-30T12:55:39.828 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_validate_names": "true", 2024-03-30T12:55:39.828 INFO:teuthology.orchestra.run.smithi067.stdout: "rbd_validate_pool": "true", 2024-03-30T12:55:39.828 INFO:teuthology.orchestra.run.smithi067.stdout: "restapi_base_url": "", 2024-03-30T12:55:39.828 INFO:teuthology.orchestra.run.smithi067.stdout: "restapi_log_level": "", 2024-03-30T12:55:39.828 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_acl_grants_max_num": "100", 2024-03-30T12:55:39.828 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_admin_entry": "admin", 2024-03-30T12:55:39.828 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-03-30T12:55:39.829 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_backend_store": "rados", 2024-03-30T12:55:39.829 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_barbican_url": "", 2024-03-30T12:55:39.829 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_beast_enable_async": "true", 2024-03-30T12:55:39.829 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_bucket_counters_cache": "false", 2024-03-30T12:55:39.829 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-03-30T12:55:39.829 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-03-30T12:55:39.829 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-03-30T12:55:39.829 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_bucket_index_max_aio": "128", 2024-03-30T12:55:39.829 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-03-30T12:55:39.829 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-03-30T12:55:39.829 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_bucket_quota_ttl": "600", 2024-03-30T12:55:39.829 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-03-30T12:55:39.829 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_cache_enabled": "true", 2024-03-30T12:55:39.829 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_cache_expiry_interval": "900", 2024-03-30T12:55:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_cache_lru_size": "10000", 2024-03-30T12:55:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_config_store": "rados", 2024-03-30T12:55:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_content_length_compat": "false", 2024-03-30T12:55:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_copy_obj_progress": "true", 2024-03-30T12:55:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-03-30T12:55:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_cors_rules_max_num": "100", 2024-03-30T12:55:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_cross_domain_policy": "", 2024-03-30T12:55:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_default_encryption_key": "", 2024-03-30T12:55:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_kmip_addr": "", 2024-03-30T12:55:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_kmip_ca_path": "", 2024-03-30T12:55:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_kmip_client_cert": "", 2024-03-30T12:55:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_kmip_client_key": "", 2024-03-30T12:55:39.830 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-03-30T12:55:39.831 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_kmip_password": "", 2024-03-30T12:55:39.831 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-03-30T12:55:39.831 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_kmip_username": "", 2024-03-30T12:55:39.831 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_require_ssl": "true", 2024-03-30T12:55:39.831 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-03-30T12:55:39.831 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-03-30T12:55:39.831 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-03-30T12:55:39.831 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-03-30T12:55:39.831 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-03-30T12:55:39.831 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-03-30T12:55:39.831 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-03-30T12:55:39.831 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-03-30T12:55:39.831 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-03-30T12:55:39.831 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-03-30T12:55:39.831 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-03-30T12:55:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-03-30T12:55:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-03-30T12:55:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-03-30T12:55:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_suppress_logs": "true", 2024-03-30T12:55:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_addr": "", 2024-03-30T12:55:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_auth": "token", 2024-03-30T12:55:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_namespace": "", 2024-03-30T12:55:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_prefix": "", 2024-03-30T12:55:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-03-30T12:55:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-03-30T12:55:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-03-30T12:55:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-03-30T12:55:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_token_file": "", 2024-03-30T12:55:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-03-30T12:55:39.832 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_curl_buffersize": "524288", 2024-03-30T12:55:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_curl_low_speed_limit": "1024", 2024-03-30T12:55:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_curl_low_speed_time": "300", 2024-03-30T12:55:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_curl_tcp_keepalive": "0", 2024-03-30T12:55:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-03-30T12:55:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-03-30T12:55:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-03-30T12:55:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-03-30T12:55:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-03-30T12:55:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d3n_l1_fadvise": "4", 2024-03-30T12:55:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-03-30T12:55:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-03-30T12:55:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-03-30T12:55:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d4n_host": "127.0.0.1", 2024-03-30T12:55:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_d4n_port": "6379", 2024-03-30T12:55:39.833 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-03-30T12:55:39.834 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_data_log_changes_size": "1000", 2024-03-30T12:55:39.834 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_data_log_num_shards": "128", 2024-03-30T12:55:39.834 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-03-30T12:55:39.834 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_data_log_window": "30", 2024-03-30T12:55:39.834 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_data_notify_interval_msec": "0", 2024-03-30T12:55:39.834 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_data_sync_poll_interval": "20", 2024-03-30T12:55:39.834 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_data_sync_spawn_window": "20", 2024-03-30T12:55:39.834 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-03-30T12:55:39.834 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-03-30T12:55:39.834 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-03-30T12:55:39.834 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_default_data_log_backing": "fifo", 2024-03-30T12:55:39.834 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-03-30T12:55:39.834 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_default_region_info_oid": "default.region", 2024-03-30T12:55:39.834 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-03-30T12:55:39.835 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-03-30T12:55:39.835 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_defer_to_bucket_acls": "", 2024-03-30T12:55:39.835 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-03-30T12:55:39.835 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-03-30T12:55:39.835 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-03-30T12:55:39.835 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-03-30T12:55:39.835 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-03-30T12:55:39.835 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-03-30T12:55:39.835 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-03-30T12:55:39.835 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-03-30T12:55:39.835 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_data_res": "500.000000", 2024-03-30T12:55:39.835 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-03-30T12:55:39.835 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-03-30T12:55:39.835 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-03-30T12:55:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-03-30T12:55:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dns_name": "", 2024-03-30T12:55:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dns_s3website_name": "", 2024-03-30T12:55:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_dynamic_resharding": "true", 2024-03-30T12:55:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-03-30T12:55:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_enable_gc_threads": "true", 2024-03-30T12:55:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_enable_lc_threads": "true", 2024-03-30T12:55:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_enable_ops_log": "false", 2024-03-30T12:55:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_enable_quota_threads": "true", 2024-03-30T12:55:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_enable_static_website": "false", 2024-03-30T12:55:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_enable_usage_log": "false", 2024-03-30T12:55:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_enforce_swift_acls": "true", 2024-03-30T12:55:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_exit_timeout_secs": "120", 2024-03-30T12:55:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_expose_bucket": "false", 2024-03-30T12:55:39.836 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_extended_http_attrs": "", 2024-03-30T12:55:39.837 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_filter": "none", 2024-03-30T12:55:39.837 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-03-30T12:55:39.837 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_frontends": "beast port=7480", 2024-03-30T12:55:39.837 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_gc_max_concurrent_io": "10", 2024-03-30T12:55:39.837 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_gc_max_deferred": "50", 2024-03-30T12:55:39.837 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-03-30T12:55:39.837 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_gc_max_objs": "32", 2024-03-30T12:55:39.837 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_gc_max_queue_size": "134213632", 2024-03-30T12:55:39.837 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_gc_max_trim_chunk": "16", 2024-03-30T12:55:39.837 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_gc_obj_min_wait": "7200", 2024-03-30T12:55:39.837 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_gc_processor_max_time": "3600", 2024-03-30T12:55:39.837 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_gc_processor_period": "3600", 2024-03-30T12:55:39.837 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-03-30T12:55:39.837 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_get_obj_window_size": "16777216", 2024-03-30T12:55:39.838 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_healthcheck_disabling_path": "", 2024-03-30T12:55:39.838 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ignore_get_invalid_range": "false", 2024-03-30T12:55:39.838 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_init_timeout": "300", 2024-03-30T12:55:39.838 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-03-30T12:55:39.838 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-03-30T12:55:39.838 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_kafka_connection_idle": "30", 2024-03-30T12:55:39.838 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_kafka_sleep_timeout": "10", 2024-03-30T12:55:39.838 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-03-30T12:55:39.838 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-03-30T12:55:39.838 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-03-30T12:55:39.838 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_admin_domain": "", 2024-03-30T12:55:39.838 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_admin_password": "", 2024-03-30T12:55:39.838 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_admin_password_path": "", 2024-03-30T12:55:39.838 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_admin_project": "", 2024-03-30T12:55:39.839 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_admin_tenant": "", 2024-03-30T12:55:39.839 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_admin_token": "", 2024-03-30T12:55:39.839 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_admin_token_path": "", 2024-03-30T12:55:39.839 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_admin_user": "", 2024-03-30T12:55:39.839 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_api_version": "2", 2024-03-30T12:55:39.839 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_barbican_domain": "", 2024-03-30T12:55:39.839 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_barbican_password": "", 2024-03-30T12:55:39.839 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_barbican_project": "", 2024-03-30T12:55:39.839 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_barbican_tenant": "", 2024-03-30T12:55:39.839 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_barbican_user": "", 2024-03-30T12:55:39.839 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-03-30T12:55:39.839 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_implicit_tenants": "false", 2024-03-30T12:55:39.839 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-03-30T12:55:39.839 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_service_token_enabled": "false", 2024-03-30T12:55:39.839 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_token_cache_size": "10000", 2024-03-30T12:55:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_url": "", 2024-03-30T12:55:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_keystone_verify_ssl": "true", 2024-03-30T12:55:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_lc_debug_interval": "-1", 2024-03-30T12:55:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_lc_lock_max_time": "90", 2024-03-30T12:55:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_lc_max_objs": "32", 2024-03-30T12:55:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_lc_max_rules": "1000", 2024-03-30T12:55:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_lc_max_worker": "3", 2024-03-30T12:55:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_lc_max_wp_worker": "3", 2024-03-30T12:55:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_lc_thread_delay": "0", 2024-03-30T12:55:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-03-30T12:55:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ldap_dnattr": "uid", 2024-03-30T12:55:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-03-30T12:55:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ldap_searchfilter": "", 2024-03-30T12:55:39.840 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-03-30T12:55:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ldap_uri": "ldaps://", 2024-03-30T12:55:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-03-30T12:55:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-03-30T12:55:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-03-30T12:55:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_log_http_headers": "", 2024-03-30T12:55:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_log_nonexistent_bucket": "false", 2024-03-30T12:55:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-03-30T12:55:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_log_object_name_utc": "false", 2024-03-30T12:55:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-03-30T12:55:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-03-30T12:55:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_attr_name_len": "0", 2024-03-30T12:55:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_attr_size": "0", 2024-03-30T12:55:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_attrs_num_in_req": "0", 2024-03-30T12:55:39.841 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_chunk_size": "4194304", 2024-03-30T12:55:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_concurrent_requests": "1024", 2024-03-30T12:55:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-03-30T12:55:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_dynamic_shards": "1999", 2024-03-30T12:55:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_listing_results": "1000", 2024-03-30T12:55:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_notify_retries": "10", 2024-03-30T12:55:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_objs_per_shard": "100000", 2024-03-30T12:55:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_put_param_size": "1048576", 2024-03-30T12:55:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_put_size": "5368709120", 2024-03-30T12:55:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_max_slo_entries": "1000", 2024-03-30T12:55:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_md_log_max_shards": "64", 2024-03-30T12:55:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_md_notify_interval_msec": "200", 2024-03-30T12:55:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_meta_sync_poll_interval": "20", 2024-03-30T12:55:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_meta_sync_spawn_window": "20", 2024-03-30T12:55:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-03-30T12:55:39.842 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_mp_lock_max_time": "600", 2024-03-30T12:55:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-03-30T12:55:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_multipart_min_part_size": "5242880", 2024-03-30T12:55:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-03-30T12:55:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-03-30T12:55:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_fhcache_size": "2017", 2024-03-30T12:55:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-03-30T12:55:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-03-30T12:55:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_lru_lanes": "5", 2024-03-30T12:55:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_max_gc": "300", 2024-03-30T12:55:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-03-30T12:55:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_run_gc_threads": "false", 2024-03-30T12:55:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_run_lc_threads": "false", 2024-03-30T12:55:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_run_quota_threads": "false", 2024-03-30T12:55:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_run_sync_thread": "false", 2024-03-30T12:55:39.843 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-03-30T12:55:39.844 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-03-30T12:55:39.844 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_num_async_rados_threads": "32", 2024-03-30T12:55:39.844 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_num_control_oids": "8", 2024-03-30T12:55:39.844 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_numa_node": "-1", 2024-03-30T12:55:39.844 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_obj_stripe_size": "4194304", 2024-03-30T12:55:39.844 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-03-30T12:55:39.844 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_objexp_chunk_size": "100", 2024-03-30T12:55:39.844 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_objexp_gc_interval": "600", 2024-03-30T12:55:39.844 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_objexp_hints_num_shards": "127", 2024-03-30T12:55:39.844 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-03-30T12:55:39.844 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-03-30T12:55:39.844 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_op_thread_timeout": "600", 2024-03-30T12:55:39.844 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_op_tracing": "false", 2024-03-30T12:55:39.844 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_opa_token": "", 2024-03-30T12:55:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_opa_url": "", 2024-03-30T12:55:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_opa_verify_ssl": "true", 2024-03-30T12:55:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-03-30T12:55:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-03-30T12:55:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ops_log_rados": "false", 2024-03-30T12:55:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_ops_log_socket_path": "", 2024-03-30T12:55:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-03-30T12:55:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-03-30T12:55:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-03-30T12:55:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_period_push_interval": "2.000000", 2024-03-30T12:55:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_period_push_interval_max": "30.000000", 2024-03-30T12:55:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_period_root_pool": ".rgw.root", 2024-03-30T12:55:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-03-30T12:55:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-03-30T12:55:39.845 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_posix_cache_lanes": "3", 2024-03-30T12:55:39.846 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-03-30T12:55:39.846 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_posix_cache_max_buckets": "100", 2024-03-30T12:55:39.846 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_posix_cache_partitions": "3", 2024-03-30T12:55:39.846 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-03-30T12:55:39.846 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_print_continue": "true", 2024-03-30T12:55:39.846 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_print_prohibited_content_length": "false", 2024-03-30T12:55:39.846 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-03-30T12:55:39.846 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-03-30T12:55:39.846 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-03-30T12:55:39.846 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-03-30T12:55:39.846 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_rados_tracing": "false", 2024-03-30T12:55:39.846 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_realm": "", 2024-03-30T12:55:39.846 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_realm_id": "", 2024-03-30T12:55:39.846 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-03-30T12:55:39.847 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_region": "", 2024-03-30T12:55:39.847 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_region_root_pool": ".rgw.root", 2024-03-30T12:55:39.847 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_relaxed_region_enforcement": "false", 2024-03-30T12:55:39.847 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-03-30T12:55:39.847 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-03-30T12:55:39.847 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_request_uri": "", 2024-03-30T12:55:39.847 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_reshard_batch_size": "64", 2024-03-30T12:55:39.847 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-03-30T12:55:39.847 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_reshard_max_aio": "128", 2024-03-30T12:55:39.847 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_reshard_num_logs": "16", 2024-03-30T12:55:39.847 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_reshard_thread_interval": "600", 2024-03-30T12:55:39.847 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_resolve_cname": "false", 2024-03-30T12:55:39.847 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_rest_getusage_op_compat": "false", 2024-03-30T12:55:39.847 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_run_sync_thread": "true", 2024-03-30T12:55:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-03-30T12:55:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_s3_auth_use_keystone": "false", 2024-03-30T12:55:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_s3_auth_use_ldap": "false", 2024-03-30T12:55:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_s3_auth_use_rados": "true", 2024-03-30T12:55:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_s3_auth_use_sts": "false", 2024-03-30T12:55:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-03-30T12:55:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_s3_success_create_obj_status": "0", 2024-03-30T12:55:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-03-30T12:55:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_scheduler_type": "throttler", 2024-03-30T12:55:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_script_uri": "", 2024-03-30T12:55:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_service_provider_name": "", 2024-03-30T12:55:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-03-30T12:55:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sts_client_id": "", 2024-03-30T12:55:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sts_client_secret": "", 2024-03-30T12:55:39.848 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sts_entry": "sts", 2024-03-30T12:55:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sts_key": "sts", 2024-03-30T12:55:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sts_max_session_duration": "43200", 2024-03-30T12:55:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sts_min_session_duration": "900", 2024-03-30T12:55:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sts_token_introspection_url": "", 2024-03-30T12:55:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_account_in_url": "false", 2024-03-30T12:55:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_auth_entry": "auth", 2024-03-30T12:55:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_auth_url": "", 2024-03-30T12:55:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_custom_header": "", 2024-03-30T12:55:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_enforce_content_length": "false", 2024-03-30T12:55:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_need_stats": "true", 2024-03-30T12:55:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_tenant_name": "", 2024-03-30T12:55:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_token_expiration": "86400", 2024-03-30T12:55:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_url": "", 2024-03-30T12:55:39.849 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_url_prefix": "swift", 2024-03-30T12:55:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_swift_versioning_enabled": "false", 2024-03-30T12:55:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-03-30T12:55:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-03-30T12:55:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_lease_period": "120", 2024-03-30T12:55:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-03-30T12:55:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_log_trim_interval": "1200", 2024-03-30T12:55:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-03-30T12:55:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-03-30T12:55:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-03-30T12:55:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_obj_etag_verify": "false", 2024-03-30T12:55:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_trace_history_size": "4096", 2024-03-30T12:55:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-03-30T12:55:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-03-30T12:55:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_thread_pool_size": "512", 2024-03-30T12:55:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_topic_persistency_max_retries": "0", 2024-03-30T12:55:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-03-30T12:55:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-03-30T12:55:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_topic_require_publish_policy": "false", 2024-03-30T12:55:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_torrent_comment": "", 2024-03-30T12:55:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_torrent_createby": "", 2024-03-30T12:55:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_torrent_encoding": "", 2024-03-30T12:55:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_torrent_flag": "false", 2024-03-30T12:55:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_torrent_max_size": "5368709120", 2024-03-30T12:55:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_torrent_origin": "", 2024-03-30T12:55:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_torrent_sha_unit": "524288", 2024-03-30T12:55:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_torrent_tracker": "", 2024-03-30T12:55:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_trust_forwarded_https": "false", 2024-03-30T12:55:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-03-30T12:55:39.851 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_usage_log_tick_interval": "30", 2024-03-30T12:55:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_usage_max_shards": "32", 2024-03-30T12:55:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_usage_max_user_shards": "1", 2024-03-30T12:55:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_use_opa_authz": "false", 2024-03-30T12:55:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_counters_cache": "false", 2024-03-30T12:55:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_counters_cache_size": "10000", 2024-03-30T12:55:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-03-30T12:55:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_default_quota_max_size": "-1", 2024-03-30T12:55:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_max_buckets": "1000", 2024-03-30T12:55:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_policies_max_num": "100", 2024-03-30T12:55:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-03-30T12:55:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-03-30T12:55:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_quota_sync_interval": "86400", 2024-03-30T12:55:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-03-30T12:55:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_user_unique_email": "true", 2024-03-30T12:55:39.852 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_verify_ssl": "true", 2024-03-30T12:55:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_website_routing_rules_max_num": "50", 2024-03-30T12:55:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_zone": "", 2024-03-30T12:55:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_zone_id": "", 2024-03-30T12:55:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-03-30T12:55:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_zonegroup": "", 2024-03-30T12:55:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_zonegroup_id": "", 2024-03-30T12:55:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-03-30T12:55:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: "rgwlc_auto_session_clear": "true", 2024-03-30T12:55:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: "rgwlc_skip_bucket_step": "false", 2024-03-30T12:55:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_block_size": "4096", 2024-03-30T12:55:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-03-30T12:55:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-03-30T12:55:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-03-30T12:55:39.853 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-03-30T12:55:39.854 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_cache_shard_bits": "4", 2024-03-30T12:55:39.854 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_cache_size": "536870912", 2024-03-30T12:55:39.854 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_cache_type": "binned_lru", 2024-03-30T12:55:39.854 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-03-30T12:55:39.854 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-03-30T12:55:39.854 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-03-30T12:55:39.854 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_collect_compaction_stats": "false", 2024-03-30T12:55:39.854 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_collect_extended_stats": "false", 2024-03-30T12:55:39.854 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_collect_memory_stats": "false", 2024-03-30T12:55:39.854 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-03-30T12:55:39.854 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_index_type": "binary_search", 2024-03-30T12:55:39.854 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_log_to_ceph_log": "true", 2024-03-30T12:55:39.854 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_metadata_block_size": "4096", 2024-03-30T12:55:39.854 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_partition_filters": "false", 2024-03-30T12:55:39.854 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_perf": "false", 2024-03-30T12:55:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-03-30T12:55:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-03-30T12:55:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: "rotating_keys_renewal_timeout": "10", 2024-03-30T12:55:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: "run_dir": "/var/run/ceph", 2024-03-30T12:55:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_block_create": "true", 2024-03-30T12:55:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_cache_lru_size": "67108864", 2024-03-30T12:55:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_cbjournal_size": "5368709120", 2024-03-30T12:55:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_data_delta_based_overwrite": "0", 2024-03-30T12:55:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_default_max_object_size": "16777216", 2024-03-30T12:55:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-03-30T12:55:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_device_size": "53687091200", 2024-03-30T12:55:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_journal_batch_capacity": "16", 2024-03-30T12:55:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-03-30T12:55:39.855 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-03-30T12:55:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_journal_iodepth_limit": "5", 2024-03-30T12:55:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_main_device_type": "SSD", 2024-03-30T12:55:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_max_concurrent_transactions": "8", 2024-03-30T12:55:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-03-30T12:55:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-03-30T12:55:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-03-30T12:55:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-03-30T12:55:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: "seastore_segment_size": "67108864", 2024-03-30T12:55:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: "set_keepcaps": "false", 2024-03-30T12:55:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: "setgroup": "", 2024-03-30T12:55:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: "setuser": "", 2024-03-30T12:55:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: "setuser_match_path": "", 2024-03-30T12:55:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: "target_max_misplaced_ratio": "0.050000", 2024-03-30T12:55:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: "thp": "false", 2024-03-30T12:55:39.856 INFO:teuthology.orchestra.run.smithi067.stdout: "threadpool_default_timeout": "60", 2024-03-30T12:55:39.857 INFO:teuthology.orchestra.run.smithi067.stdout: "threadpool_empty_queue_max_wait": "2", 2024-03-30T12:55:39.857 INFO:teuthology.orchestra.run.smithi067.stdout: "throttler_perf_counter": "true" 2024-03-30T12:55:39.857 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-30T12:55:39.859 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-03-30T12:55:39.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:39.981+0000 7fd47dd0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:39.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:39.989+0000 7fd47dd0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:40.011 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a: {} 2024-03-30T12:55:40.011 INFO:teuthology.orchestra.run.smithi067.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-30T12:55:40.028 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b: {} 2024-03-30T12:55:40.028 INFO:teuthology.orchestra.run.smithi067.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-30T12:55:40.041 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c: {} 2024-03-30T12:55:40.041 INFO:teuthology.orchestra.run.smithi067.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-30T12:55:40.058 DEBUG:teuthology.orchestra.run.smithi082:> type ceph-objectstore-tool 2024-03-30T12:55:40.064 INFO:teuthology.orchestra.run.smithi082.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-03-30T12:55:40.064 DEBUG:teuthology.orchestra.run.smithi067:> type ceph-objectstore-tool 2024-03-30T12:55:40.068 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-03-30T12:55:40.068 INFO:tasks.thrashosds.thrasher:Unable to test ceph-objectstore-tool, not available on all OSD nodes 2024-03-30T12:55:40.068 INFO:teuthology.run_tasks:Running task rados... 2024-03-30T12:55:40.080 INFO:tasks.rados:Beginning rados... 2024-03-30T12:55:40.080 INFO:tasks.rados:config is {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_use_overwrites': True, 'objects': 50, 'op_weights': {'copy_from': 50, 'delete': 50, 'read': 100, 'rollback': 50, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000} 2024-03-30T12:55:40.081 INFO:tasks.rados:overrides is {'admin_socket': {'branch': 'wip-ceph_test_rados-partial-reads'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': '*', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 2, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'create_rbd_pool': False, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'sha1': '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc'}}, 'rados': {'wait_for_all_active_clean_pgs': True}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5, 'disable_objectstore_tool_tests': True}, 'workunit': {'branch': 'wip-ceph_test_rados-partial-reads', 'sha1': '5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc'}} 2024-03-30T12:55:40.081 INFO:tasks.rados:config is {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_use_overwrites': True, 'objects': 50, 'op_weights': {'copy_from': 50, 'delete': 50, 'read': 100, 'rollback': 50, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000, 'wait_for_all_active_clean_pgs': True} 2024-03-30T12:55:40.081 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-03-30T12:55:40.121 INFO:tasks.rados:joining rados 2024-03-30T12:55:40.121 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-03-30T12:55:40.121 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T12:55:40.121 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-30T12:55:40.121 INFO:tasks.thrashosds.thrasher:test_pool_min_size 2024-03-30T12:55:40.122 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-03-30T12:55:40.122 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-03-30T12:55:40.122 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T12:55:40.122 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-03-30T12:55:40.123 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-30T12:55:40.123 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-03-30T12:55:40.123 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-03-30T12:55:40.123 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2024-03-30T12:55:40.124 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set teuthologyprofile k=2 m=1 crush-failure-domain=osd 2024-03-30T12:55:40.125 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T12:55:40.121+0000 7f54d0f4e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25195) UID: 0 2024-03-30T12:55:40.223 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-03-30T12:55:40.225+0000 7f605665e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25193) UID: 0 2024-03-30T12:55:40.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:40.229+0000 7fa2c3e16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:40.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:40.229+0000 7f57c6381640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:40.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:40.229+0000 7f4a6c628640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:40.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:40.237+0000 7fa2c3e16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:40.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:40.241+0000 7f57c6381640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:40.255 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-30T12:55:40.255 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-30T12:55:40.256 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-03-30T12:55:40.256 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-30T12:55:40.267 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-03-30T12:55:40.268 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T12:55:40.269 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-30T12:55:40.288 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-03-30T12:55:40.288 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T12:55:40.308 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-03-30T12:55:40.309 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T12:55:40.325 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:55:40.327+0000 7f23bea0b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25277) UID: 0 2024-03-30T12:55:40.330 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-03-30T12:55:40.330 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T12:55:40.354 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-03-30T12:55:40.354 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T12:55:40.379 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-03-30T12:55:40.380 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T12:55:40.396 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:40.396 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:40.404 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-03-30T12:55:40.404 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T12:55:40.409 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-30T12:55:40.425 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T12:55:40.423+0000 7f4c397b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25271) UID: 0 2024-03-30T12:55:40.431 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-03-30T12:55:40.431 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T12:55:40.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:40.445+0000 7f4a6a3c4640 0 --2- 172.21.15.67:0/3175635773 >> v2:172.21.15.67:3300/0 conn(0x7f4a640704a0 0x7f4a640af570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-03-30T12:55:40.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:40.449+0000 7f4a6c628640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:40.526 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-03-30T12:55:40.525+0000 7f605665e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25193) UID: 0 2024-03-30T12:55:40.541 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:40.541 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:40.554 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-30T12:55:40.644 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-03-30T12:55:40.645+0000 7f710ac50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25189) UID: 0 2024-03-30T12:55:40.714 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:40.715 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:40.728 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-30T12:55:40.728 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:55:40.727+0000 7f23bea0b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25277) UID: 0 2024-03-30T12:55:40.828 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T12:55:40.827+0000 7f4c397b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25271) UID: 0 2024-03-30T12:55:40.894 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is set 2024-03-30T12:55:40.901 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:40.901 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:40.915 INFO:tasks.rados:starting run 0 out of 1 2024-03-30T12:55:40.916 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-03-30T12:55:40.916 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure teuthologyprofile 2024-03-30T12:55:40.918 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-30T12:55:40.929 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T12:55:40.927+0000 7f4c397b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25271) UID: 0 2024-03-30T12:55:41.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:41.017+0000 7f9c87995640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:41.030 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T12:55:41.029+0000 7fbc9da50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25191) UID: 0 2024-03-30T12:55:41.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:41.037+0000 7f9c87995640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:41.071 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:41.072 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:41.085 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-30T12:55:41.171 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-03-30T12:55:41.173+0000 7f710ac50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25189) UID: 0 2024-03-30T12:55:41.232 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T12:55:41.233+0000 7fbc9da50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25191) UID: 0 2024-03-30T12:55:41.256 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:41.256 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:41.269 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-30T12:55:41.333 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:55:41.331+0000 7f23bea0b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25277) UID: 0 2024-03-30T12:55:41.433 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:41.434 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:41.434 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-03-30T12:55:41.433+0000 7f710ac50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25189) UID: 0 2024-03-30T12:55:41.446 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-30T12:55:41.535 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T12:55:41.537+0000 7fbc9da50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25191) UID: 0 2024-03-30T12:55:41.609 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:41.610 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:41.624 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-30T12:55:41.668 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T12:55:41.669+0000 7f54d0f4e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25195) UID: 0 2024-03-30T12:55:41.737 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T12:55:41.735+0000 7f75d16be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25273) UID: 0 2024-03-30T12:55:41.786 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:41.786 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:41.800 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-30T12:55:41.838 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T12:55:41.839+0000 7f75d16be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25273) UID: 0 2024-03-30T12:55:41.924 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:41.924 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:41.937 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-30T12:55:41.938 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:55:41.939+0000 7f23bea0b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25277) UID: 0 2024-03-30T12:55:42.040 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T12:55:42.039+0000 7f4c397b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25271) UID: 0 2024-03-30T12:55:42.058 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:42.058 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:42.072 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-30T12:55:42.141 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-03-30T12:55:42.141+0000 7f710ac50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25189) UID: 0 2024-03-30T12:55:42.205 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:42.205 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:42.220 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-30T12:55:42.264 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T12:55:42.263+0000 7f3b265f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25275) UID: 0 2024-03-30T12:55:42.342 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:55:42.339+0000 7f23bea0b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25277) UID: 0 2024-03-30T12:55:42.343 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:42.343 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:42.357 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-30T12:55:42.444 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:55:42.443+0000 7f23bea0b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25277) UID: 0 2024-03-30T12:55:42.451 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T12:55:42.525 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:42.526 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:42.540 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-30T12:55:42.543 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T12:55:42.545+0000 7fbc9da50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25191) UID: 0 2024-03-30T12:55:42.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:42.577+0000 7f73d343f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:42.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:42.585+0000 7f73d343f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:42.615 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-03-30T12:55:42.615 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T12:55:42.638 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-03-30T12:55:42.638 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T12:55:42.645 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T12:55:42.643+0000 7f75d16be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25273) UID: 0 2024-03-30T12:55:42.662 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-03-30T12:55:42.662 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T12:55:42.689 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-03-30T12:55:42.689 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T12:55:42.705 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:42.705 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:42.719 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-03-30T12:55:42.719 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T12:55:42.720 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-30T12:55:42.745 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-03-30T12:55:42.745 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T12:55:42.764 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T12:55:42.763+0000 7f3b265f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25275) UID: 0 2024-03-30T12:55:42.774 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-03-30T12:55:42.774 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T12:55:42.803 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-03-30T12:55:42.803 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T12:55:42.846 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T12:55:42.843+0000 7f3b265f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25275) UID: 0 2024-03-30T12:55:42.883 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T12:55:42.884 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T12:55:42.884 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-03-30T12:55:42.884 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-03-30T12:55:42.884 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T12:55:42.898 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-30T12:55:42.914 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'unique_pool_0' created 2024-03-30T12:55:42.916 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-03-30T12:55:42.938 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 allow_ec_overwrites true 2024-03-30T12:55:42.979 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T12:55:42.981+0000 7f54d0f4e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25195) UID: 0 2024-03-30T12:55:43.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:43.029+0000 7fa6ec7ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:43.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:43.045+0000 7fa6ec7ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:43.047 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-03-30T12:55:43.049+0000 7f710ac50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25189) UID: 0 2024-03-30T12:55:43.062 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T12:55:43.062 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-03-30T12:55:43.062 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-03-30T12:55:43.062 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [] 2024-03-30T12:55:43.062 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T12:55:43.076 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-30T12:55:43.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:43.085+0000 7f2f4216e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:43.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:43.101+0000 7f2f4216e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:43.149 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T12:55:43.149+0000 7f54d0f4e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25195) UID: 0 2024-03-30T12:55:43.202 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T12:55:43.202 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T12:55:43.202 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T12:55:43.202 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T12:55:43.216 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-30T12:55:43.260 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T12:55:43.259+0000 7f75d16be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25273) UID: 0 2024-03-30T12:55:43.331 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T12:55:43.331 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T12:55:43.332 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-03-30T12:55:43.332 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-03-30T12:55:43.332 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T12:55:43.347 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-30T12:55:43.352 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T12:55:43.353+0000 7fbc9da50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25191) UID: 0 2024-03-30T12:55:43.453 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-03-30T12:55:43.453+0000 7f605665e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25193) UID: 0 2024-03-30T12:55:43.507 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T12:55:43.507 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-03-30T12:55:43.507 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-03-30T12:55:43.508 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [] 2024-03-30T12:55:43.508 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T12:55:43.521 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-30T12:55:43.568 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:55:43.567+0000 7f23bea0b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25277) UID: 0 2024-03-30T12:55:43.639 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T12:55:43.639 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T12:55:43.639 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T12:55:43.639 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T12:55:43.655 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-30T12:55:43.655 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T12:55:43.651+0000 7f3b265f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25275) UID: 0 2024-03-30T12:55:43.755 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T12:55:43.751+0000 7f3b265f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25275) UID: 0 2024-03-30T12:55:43.820 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T12:55:43.820 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T12:55:43.820 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-03-30T12:55:43.820 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-03-30T12:55:43.820 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T12:55:43.834 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-30T12:55:43.876 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T12:55:43.875+0000 7f4c397b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25271) UID: 0 2024-03-30T12:55:43.908 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is set 2024-03-30T12:55:43.912 INFO:teuthology.orchestra.run.smithi067.stderr:set pool 1 allow_ec_overwrites to true 2024-03-30T12:55:43.948 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-03-30T12:55:43.957 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:55:43.955+0000 7f23bea0b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25277) UID: 0 2024-03-30T12:55:44.010 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T12:55:44.010 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-03-30T12:55:44.010 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-03-30T12:55:44.010 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [] 2024-03-30T12:55:44.010 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T12:55:44.024 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-03-30T12:55:44.079 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-03-30T12:55:44.081+0000 7f710ac50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25189) UID: 0 2024-03-30T12:55:44.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:44.109+0000 7fdb2c24e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:44.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:44.129+0000 7fdb2c24e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:44.156 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-30T12:55:44.157 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-30T12:55:44.157 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-03-30T12:55:44.157 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-30T12:55:44.161 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-03-30T12:55:44.157+0000 7f710ac50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25189) UID: 0 2024-03-30T12:55:44.170 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-03-30T12:55:44.263 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T12:55:44.265+0000 7f54d0f4e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25195) UID: 0 2024-03-30T12:55:44.334 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-30T12:55:44.334 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-30T12:55:44.334 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-03-30T12:55:44.334 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-03-30T12:55:44.334 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-30T12:55:44.348 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-03-30T12:55:44.362 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T12:55:44.363+0000 7f4c397b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25271) UID: 0 2024-03-30T12:55:44.462 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:55:44.459+0000 7f23bea0b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25277) UID: 0 2024-03-30T12:55:44.514 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-30T12:55:44.514 INFO:teuthology.orchestra.run.smithi067.stdout: "size": 20, 2024-03-30T12:55:44.514 INFO:teuthology.orchestra.run.smithi067.stdout: "duration": 600, 2024-03-30T12:55:44.514 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [] 2024-03-30T12:55:44.514 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-30T12:55:44.527 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-30T12:55:44.563 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T12:55:44.563+0000 7f4c397b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25271) UID: 0 2024-03-30T12:55:44.653 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-30T12:55:44.653 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-30T12:55:44.653 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-03-30T12:55:44.653 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-30T12:55:44.664 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T12:55:44.665+0000 7f54d0f4e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25195) UID: 0 2024-03-30T12:55:44.667 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-03-30T12:55:44.764 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-03-30T12:55:44.765+0000 7f605665e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25193) UID: 0 2024-03-30T12:55:44.827 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-30T12:55:44.840 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-30T12:55:44.840 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-30T12:55:44.840 INFO:teuthology.orchestra.run.smithi067.stdout: "complaint_time": 30, 2024-03-30T12:55:44.840 INFO:teuthology.orchestra.run.smithi067.stdout: "num_blocked_ops": 0 2024-03-30T12:55:44.840 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-30T12:55:44.856 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-03-30T12:55:44.865 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T12:55:44.867+0000 7f4c397b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25271) UID: 0 2024-03-30T12:55:44.927 INFO:teuthology.orchestra.run.smithi067.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-03-30T12:55:44.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:44.953+0000 7ff1004b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:44.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:44.965+0000 7ff1004b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:44.965 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-03-30T12:55:44.965+0000 7f605665e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25193) UID: 0 2024-03-30T12:55:44.994 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-03-30T12:55:44.994 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T12:55:45.016 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-03-30T12:55:45.016 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T12:55:45.022 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:45.022 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:45.035 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-03-30T12:55:45.036 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-03-30T12:55:45.037 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T12:55:45.057 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-03-30T12:55:45.057 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T12:55:45.066 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:55:45.063+0000 7f23bea0b640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25277) UID: 0 2024-03-30T12:55:45.096 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-03-30T12:55:45.096 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T12:55:45.122 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-03-30T12:55:45.123 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T12:55:45.146 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-03-30T12:55:45.147 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T12:55:45.158 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:45.158 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:45.167 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T12:55:45.169+0000 7f54d0f4e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25195) UID: 0 2024-03-30T12:55:45.171 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-03-30T12:55:45.173 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-03-30T12:55:45.174 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T12:55:45.268 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T12:55:45.269+0000 7f54d0f4e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25195) UID: 0 2024-03-30T12:55:45.289 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:45.290 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:45.303 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-03-30T12:55:45.392 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T12:55:45.393+0000 7fbc9da50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25191) UID: 0 2024-03-30T12:55:45.469 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T12:55:45.471+0000 7f75d16be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25273) UID: 0 2024-03-30T12:55:45.475 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:45.476 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:45.490 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-30T12:55:45.570 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T12:55:45.571+0000 7f4c397b1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25271) UID: 0 2024-03-30T12:55:45.654 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:45.655 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:45.668 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-03-30T12:55:45.671 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T12:55:45.669+0000 7f54d0f4e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25195) UID: 0 2024-03-30T12:55:45.771 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T12:55:45.771+0000 7f3b265f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25275) UID: 0 2024-03-30T12:55:45.794 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:45.794 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:45.808 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-03-30T12:55:45.873 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-03-30T12:55:45.873+0000 7f605665e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25193) UID: 0 2024-03-30T12:55:45.929 INFO:teuthology.orchestra.run.smithi067.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:45.929 INFO:teuthology.orchestra.run.smithi067.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:45.944 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-30T12:55:45.946 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-30T12:55:45.953 DEBUG:teuthology.orchestra.run.smithi067:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0 2024-03-30T12:55:45.992 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T12:55:45.991+0000 7f3b265f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25275) UID: 0 2024-03-30T12:55:46.006 INFO:tasks.rados.rados.0.smithi067.stdout:adding op weight read -> 100 2024-03-30T12:55:46.006 INFO:tasks.rados.rados.0.smithi067.stdout:adding op weight write -> 50 2024-03-30T12:55:46.006 INFO:tasks.rados.rados.0.smithi067.stdout:adding op weight delete -> 50 2024-03-30T12:55:46.006 INFO:tasks.rados.rados.0.smithi067.stdout:adding op weight snap_create -> 50 2024-03-30T12:55:46.006 INFO:tasks.rados.rados.0.smithi067.stdout:adding op weight snap_remove -> 50 2024-03-30T12:55:46.006 INFO:tasks.rados.rados.0.smithi067.stdout:adding op weight rollback -> 50 2024-03-30T12:55:46.006 INFO:tasks.rados.rados.0.smithi067.stdout:adding op weight copy_from -> 50 2024-03-30T12:55:46.006 INFO:tasks.rados.rados.0.smithi067.stdout:adding op weight write_excl -> 50 2024-03-30T12:55:46.006 INFO:tasks.rados.rados.0.smithi067.stdout:ceph version 19.0.0-1864-g5dc2ea30 (5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc) squid (dev) 2024-03-30T12:55:46.007 INFO:tasks.rados.rados.0.smithi067.stdout:Configuration: 2024-03-30T12:55:46.007 INFO:tasks.rados.rados.0.smithi067.stdout: Number of operations: 4000 2024-03-30T12:55:46.007 INFO:tasks.rados.rados.0.smithi067.stdout: Number of objects: 50 2024-03-30T12:55:46.007 INFO:tasks.rados.rados.0.smithi067.stdout: Max in flight operations: 16 2024-03-30T12:55:46.007 INFO:tasks.rados.rados.0.smithi067.stdout: Object size (in bytes): 4000000 2024-03-30T12:55:46.007 INFO:tasks.rados.rados.0.smithi067.stdout: Write stride min: 400000 2024-03-30T12:55:46.007 INFO:tasks.rados.rados.0.smithi067.stdout: Write stride max: 800000 2024-03-30T12:55:46.075 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T12:55:46.071+0000 7f3b265f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25275) UID: 0 2024-03-30T12:55:46.076 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:46.077 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:46.091 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-03-30T12:55:46.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:46.097+0000 7f65ab593640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:46.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:46.117+0000 7f65ab593640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:46.176 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T12:55:46.177+0000 7fbc9da50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25191) UID: 0 2024-03-30T12:55:46.249 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:46.250 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:46.263 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-03-30T12:55:46.276 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-03-30T12:55:46.277+0000 7f605665e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25193) UID: 0 2024-03-30T12:55:46.377 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-03-30T12:55:46.377+0000 7f710ac50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25189) UID: 0 2024-03-30T12:55:46.385 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:46.385 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:46.399 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-30T12:55:46.478 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-03-30T12:55:46.477+0000 7f710ac50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25189) UID: 0 2024-03-30T12:55:46.516 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:46.516 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:46.532 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-03-30T12:55:46.580 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T12:55:46.575+0000 7f75d16be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25273) UID: 0 2024-03-30T12:55:46.655 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:46.655 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:46.669 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-03-30T12:55:46.680 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-03-30T12:55:46.681+0000 7f605665e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25193) UID: 0 2024-03-30T12:55:46.781 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T12:55:46.779+0000 7f3b265f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25275) UID: 0 2024-03-30T12:55:46.835 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:46.835 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:46.849 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-30T12:55:46.882 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T12:55:46.881+0000 7fbc9da50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25191) UID: 0 2024-03-30T12:55:46.939 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-03-30T12:55:46.944 INFO:tasks.rados.rados.0.smithi067.stdout:1: write initial oid 1 2024-03-30T12:55:46.945 INFO:tasks.rados.rados.0.smithi067.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2024-03-30T12:55:46.952 INFO:tasks.rados.rados.0.smithi067.stdout:1: writing smithi06731802-1 from 416807 to 892056 tid 1 2024-03-30T12:55:46.961 INFO:tasks.rados.rados.0.smithi067.stdout:1: writing smithi06731802-1 from 1542129 to 2016807 tid 2 2024-03-30T12:55:46.961 INFO:tasks.rados.rados.0.smithi067.stdout:2: write initial oid 2 2024-03-30T12:55:46.961 INFO:tasks.rados.rados.0.smithi067.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2024-03-30T12:55:46.968 INFO:tasks.rados.rados.0.smithi067.stdout:2: writing smithi06731802-2 from 416807 to 892056 tid 1 2024-03-30T12:55:46.978 INFO:tasks.rados.rados.0.smithi067.stdout:2: writing smithi06731802-2 from 1542129 to 2016807 tid 2 2024-03-30T12:55:46.978 INFO:tasks.rados.rados.0.smithi067.stdout:3: write initial oid 3 2024-03-30T12:55:46.979 INFO:tasks.rados.rados.0.smithi067.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2024-03-30T12:55:46.983 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T12:55:46.981+0000 7f54d0f4e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25195) UID: 0 2024-03-30T12:55:46.987 INFO:tasks.rados.rados.0.smithi067.stdout:3: writing smithi06731802-3 from 433614 to 984112 tid 1 2024-03-30T12:55:46.995 INFO:tasks.rados.rados.0.smithi067.stdout:3: writing smithi06731802-3 from 1600611 to 2033614 tid 2 2024-03-30T12:55:46.996 INFO:tasks.rados.rados.0.smithi067.stdout:4: write initial oid 4 2024-03-30T12:55:46.996 INFO:tasks.rados.rados.0.smithi067.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2024-03-30T12:55:47.004 INFO:tasks.rados.rados.0.smithi067.stdout:4: writing smithi06731802-4 from 450421 to 1076168 tid 1 2024-03-30T12:55:47.011 INFO:tasks.rados.rados.0.smithi067.stdout:4: writing smithi06731802-4 from 1659093 to 2050421 tid 2 2024-03-30T12:55:47.012 INFO:tasks.rados.rados.0.smithi067.stdout:5: write initial oid 5 2024-03-30T12:55:47.012 INFO:tasks.rados.rados.0.smithi067.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2024-03-30T12:55:47.020 INFO:tasks.rados.rados.0.smithi067.stdout:5: writing smithi06731802-5 from 467228 to 1168224 tid 1 2024-03-30T12:55:47.026 INFO:tasks.rados.rados.0.smithi067.stdout:5: writing smithi06731802-5 from 1717575 to 2067228 tid 2 2024-03-30T12:55:47.026 INFO:tasks.rados.rados.0.smithi067.stdout:6: write initial oid 6 2024-03-30T12:55:47.026 INFO:tasks.rados.rados.0.smithi067.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2024-03-30T12:55:47.029 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:47.029 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:47.034 INFO:tasks.rados.rados.0.smithi067.stdout:6: writing smithi06731802-6 from 484035 to 1260280 tid 1 2024-03-30T12:55:47.039 INFO:tasks.rados.rados.0.smithi067.stdout:6: writing smithi06731802-6 from 2059704 to 2084035 tid 2 2024-03-30T12:55:47.039 INFO:tasks.rados.rados.0.smithi067.stdout:7: write initial oid 7 2024-03-30T12:55:47.040 INFO:tasks.rados.rados.0.smithi067.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2024-03-30T12:55:47.045 INFO:tasks.rados.rados.0.smithi067.stdout:7: writing smithi06731802-7 from 500842 to 952336 tid 1 2024-03-30T12:55:47.048 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-03-30T12:55:47.053 INFO:tasks.rados.rados.0.smithi067.stdout:7: writing smithi06731802-7 from 1718186 to 2100842 tid 2 2024-03-30T12:55:47.053 INFO:tasks.rados.rados.0.smithi067.stdout:8: write initial oid 8 2024-03-30T12:55:47.053 INFO:tasks.rados.rados.0.smithi067.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2024-03-30T12:55:47.059 INFO:tasks.rados.rados.0.smithi067.stdout:8: writing smithi06731802-8 from 517649 to 1044392 tid 1 2024-03-30T12:55:47.066 INFO:tasks.rados.rados.0.smithi067.stdout:8: writing smithi06731802-8 from 1776668 to 2117649 tid 2 2024-03-30T12:55:47.066 INFO:tasks.rados.rados.0.smithi067.stdout:9: write initial oid 9 2024-03-30T12:55:47.066 INFO:tasks.rados.rados.0.smithi067.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2024-03-30T12:55:47.074 INFO:tasks.rados.rados.0.smithi067.stdout:9: writing smithi06731802-9 from 534456 to 1252801 tid 1 2024-03-30T12:55:47.078 INFO:tasks.rados.rados.0.smithi067.stdout:9: writing smithi06731802-9 from 1951503 to 2134456 tid 2 2024-03-30T12:55:47.078 INFO:tasks.rados.rados.0.smithi067.stdout:10: write initial oid 10 2024-03-30T12:55:47.078 INFO:tasks.rados.rados.0.smithi067.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2024-03-30T12:55:47.086 INFO:tasks.rados.rados.0.smithi067.stdout:10: writing smithi06731802-10 from 551263 to 1344857 tid 1 2024-03-30T12:55:47.091 INFO:tasks.rados.rados.0.smithi067.stdout:10: writing smithi06731802-10 from 1893632 to 2151263 tid 2 2024-03-30T12:55:47.091 INFO:tasks.rados.rados.0.smithi067.stdout:11: write initial oid 11 2024-03-30T12:55:47.091 INFO:tasks.rados.rados.0.smithi067.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2024-03-30T12:55:47.092 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T12:55:47.091+0000 7f3b265f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25275) UID: 0 2024-03-30T12:55:47.097 INFO:tasks.rados.rados.0.smithi067.stdout:11: writing smithi06731802-11 from 568070 to 1036913 tid 1 2024-03-30T12:55:47.103 INFO:tasks.rados.rados.0.smithi067.stdout:11: writing smithi06731802-11 from 1552114 to 2168070 tid 2 2024-03-30T12:55:47.104 INFO:tasks.rados.rados.0.smithi067.stdout:12: write initial oid 12 2024-03-30T12:55:47.104 INFO:tasks.rados.rados.0.smithi067.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2024-03-30T12:55:47.110 INFO:tasks.rados.rados.0.smithi067.stdout:12: writing smithi06731802-12 from 584877 to 1128969 tid 1 2024-03-30T12:55:47.116 INFO:tasks.rados.rados.0.smithi067.stdout:12: writing smithi06731802-12 from 1610596 to 2172599 tid 2 2024-03-30T12:55:47.116 INFO:tasks.rados.rados.0.smithi067.stdout:13: write initial oid 13 2024-03-30T12:55:47.116 INFO:tasks.rados.rados.0.smithi067.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2024-03-30T12:55:47.123 INFO:tasks.rados.rados.0.smithi067.stdout:13: writing smithi06731802-13 from 601684 to 1221025 tid 1 2024-03-30T12:55:47.129 INFO:tasks.rados.rados.0.smithi067.stdout:13: writing smithi06731802-13 from 1669078 to 2201684 tid 2 2024-03-30T12:55:47.129 INFO:tasks.rados.rados.0.smithi067.stdout:14: write initial oid 14 2024-03-30T12:55:47.129 INFO:tasks.rados.rados.0.smithi067.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2024-03-30T12:55:47.137 INFO:tasks.rados.rados.0.smithi067.stdout:14: writing smithi06731802-14 from 618491 to 1313081 tid 1 2024-03-30T12:55:47.142 INFO:tasks.rados.rados.0.smithi067.stdout:14: writing smithi06731802-14 from 2011207 to 2218491 tid 2 2024-03-30T12:55:47.142 INFO:tasks.rados.rados.0.smithi067.stdout:15: write initial oid 15 2024-03-30T12:55:47.142 INFO:tasks.rados.rados.0.smithi067.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2024-03-30T12:55:47.150 INFO:tasks.rados.rados.0.smithi067.stdout:15: writing smithi06731802-15 from 635298 to 1405137 tid 1 2024-03-30T12:55:47.156 INFO:tasks.rados.rados.0.smithi067.stdout:15: writing smithi06731802-15 from 2069689 to 2235298 tid 2 2024-03-30T12:55:47.156 INFO:tasks.rados.rados.0.smithi067.stdout:16: write initial oid 16 2024-03-30T12:55:47.156 INFO:tasks.rados.rados.0.smithi067.stdout:1: finishing write tid 1 to smithi06731802-1 2024-03-30T12:55:47.156 INFO:tasks.rados.rados.0.smithi067.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2024-03-30T12:55:47.162 INFO:tasks.rados.rados.0.smithi067.stdout:16: writing smithi06731802-16 from 652105 to 1097193 tid 1 2024-03-30T12:55:47.169 INFO:tasks.rados.rados.0.smithi067.stdout:16: writing smithi06731802-16 from 1728171 to 2181159 tid 2 2024-03-30T12:55:47.169 INFO:tasks.rados.rados.0.smithi067.stdout: waiting on 16 2024-03-30T12:55:47.169 INFO:tasks.rados.rados.0.smithi067.stdout:1: finishing write tid 2 to smithi06731802-1 2024-03-30T12:55:47.169 INFO:tasks.rados.rados.0.smithi067.stdout:1: finishing write tid 3 to smithi06731802-1 2024-03-30T12:55:47.169 INFO:tasks.rados.rados.0.smithi067.stdout:1: finishing write tid 4 to smithi06731802-1 2024-03-30T12:55:47.169 INFO:tasks.rados.rados.0.smithi067.stdout:1: oid 1 updating version 0 to 1 2024-03-30T12:55:47.169 INFO:tasks.rados.rados.0.smithi067.stdout:1: oid 1 updating version 1 to 2 2024-03-30T12:55:47.169 INFO:tasks.rados.rados.0.smithi067.stdout:1: oid 1 updating version 2 to 3 2024-03-30T12:55:47.169 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-03-30T12:55:47.169 INFO:tasks.rados.rados.0.smithi067.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-03-30T12:55:47.169 INFO:tasks.rados.rados.0.smithi067.stdout:2: finishing write tid 1 to smithi06731802-2 2024-03-30T12:55:47.169 INFO:tasks.rados.rados.0.smithi067.stdout:1: done (15 left) 2024-03-30T12:55:47.169 INFO:tasks.rados.rados.0.smithi067.stdout:17: write initial oid 17 2024-03-30T12:55:47.170 INFO:tasks.rados.rados.0.smithi067.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2024-03-30T12:55:47.176 INFO:tasks.rados.rados.0.smithi067.stdout:17: writing smithi06731802-17 from 668912 to 1305602 tid 1 2024-03-30T12:55:47.183 INFO:tasks.rados.rados.0.smithi067.stdout:17: writing smithi06731802-17 from 1903006 to 2268912 tid 2 2024-03-30T12:55:47.183 INFO:tasks.rados.rados.0.smithi067.stdout: waiting on 16 2024-03-30T12:55:47.183 INFO:tasks.rados.rados.0.smithi067.stdout:2: finishing write tid 2 to smithi06731802-2 2024-03-30T12:55:47.183 INFO:tasks.rados.rados.0.smithi067.stdout:2: finishing write tid 3 to smithi06731802-2 2024-03-30T12:55:47.183 INFO:tasks.rados.rados.0.smithi067.stdout:2: finishing write tid 4 to smithi06731802-2 2024-03-30T12:55:47.183 INFO:tasks.rados.rados.0.smithi067.stdout:2: oid 2 updating version 0 to 1 2024-03-30T12:55:47.183 INFO:tasks.rados.rados.0.smithi067.stdout:2: oid 2 updating version 1 to 2 2024-03-30T12:55:47.184 INFO:tasks.rados.rados.0.smithi067.stdout:2: oid 2 updating version 2 to 3 2024-03-30T12:55:47.184 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-03-30T12:55:47.184 INFO:tasks.rados.rados.0.smithi067.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-03-30T12:55:47.184 INFO:tasks.rados.rados.0.smithi067.stdout:3: finishing write tid 1 to smithi06731802-3 2024-03-30T12:55:47.184 INFO:tasks.rados.rados.0.smithi067.stdout:3: finishing write tid 2 to smithi06731802-3 2024-03-30T12:55:47.184 INFO:tasks.rados.rados.0.smithi067.stdout:3: finishing write tid 3 to smithi06731802-3 2024-03-30T12:55:47.184 INFO:tasks.rados.rados.0.smithi067.stdout:3: finishing write tid 4 to smithi06731802-3 2024-03-30T12:55:47.184 INFO:tasks.rados.rados.0.smithi067.stdout:3: oid 3 updating version 0 to 2 2024-03-30T12:55:47.184 INFO:tasks.rados.rados.0.smithi067.stdout:3: oid 3 version 2 is already newer than 1 2024-03-30T12:55:47.184 INFO:tasks.rados.rados.0.smithi067.stdout:3: oid 3 updating version 2 to 3 2024-03-30T12:55:47.184 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-03-30T12:55:47.184 INFO:tasks.rados.rados.0.smithi067.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-03-30T12:55:47.184 INFO:tasks.rados.rados.0.smithi067.stdout:4: finishing write tid 1 to smithi06731802-4 2024-03-30T12:55:47.184 INFO:tasks.rados.rados.0.smithi067.stdout:4: finishing write tid 2 to smithi06731802-4 2024-03-30T12:55:47.184 INFO:tasks.rados.rados.0.smithi067.stdout:4: finishing write tid 3 to smithi06731802-4 2024-03-30T12:55:47.185 INFO:tasks.rados.rados.0.smithi067.stdout:4: finishing write tid 4 to smithi06731802-4 2024-03-30T12:55:47.185 INFO:tasks.rados.rados.0.smithi067.stdout:4: oid 4 updating version 0 to 2 2024-03-30T12:55:47.185 INFO:tasks.rados.rados.0.smithi067.stdout:4: oid 4 updating version 2 to 3 2024-03-30T12:55:47.185 INFO:tasks.rados.rados.0.smithi067.stdout:4: oid 4 version 3 is already newer than 1 2024-03-30T12:55:47.185 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-03-30T12:55:47.185 INFO:tasks.rados.rados.0.smithi067.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-03-30T12:55:47.185 INFO:tasks.rados.rados.0.smithi067.stdout:5: finishing write tid 1 to smithi06731802-5 2024-03-30T12:55:47.185 INFO:tasks.rados.rados.0.smithi067.stdout:5: finishing write tid 2 to smithi06731802-5 2024-03-30T12:55:47.185 INFO:tasks.rados.rados.0.smithi067.stdout:5: finishing write tid 3 to smithi06731802-5 2024-03-30T12:55:47.185 INFO:tasks.rados.rados.0.smithi067.stdout:6: finishing write tid 1 to smithi06731802-6 2024-03-30T12:55:47.185 INFO:tasks.rados.rados.0.smithi067.stdout:6: finishing write tid 2 to smithi06731802-6 2024-03-30T12:55:47.185 INFO:tasks.rados.rados.0.smithi067.stdout:6: finishing write tid 3 to smithi06731802-6 2024-03-30T12:55:47.185 INFO:tasks.rados.rados.0.smithi067.stdout:5: finishing write tid 4 to smithi06731802-5 2024-03-30T12:55:47.185 INFO:tasks.rados.rados.0.smithi067.stdout:5: oid 5 updating version 0 to 2 2024-03-30T12:55:47.185 INFO:tasks.rados.rados.0.smithi067.stdout:5: oid 5 updating version 2 to 3 2024-03-30T12:55:47.186 INFO:tasks.rados.rados.0.smithi067.stdout:5: oid 5 version 3 is already newer than 1 2024-03-30T12:55:47.186 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-03-30T12:55:47.186 INFO:tasks.rados.rados.0.smithi067.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-03-30T12:55:47.186 INFO:tasks.rados.rados.0.smithi067.stdout:6: finishing write tid 4 to smithi06731802-6 2024-03-30T12:55:47.186 INFO:tasks.rados.rados.0.smithi067.stdout:6: oid 6 updating version 0 to 5 2024-03-30T12:55:47.186 INFO:tasks.rados.rados.0.smithi067.stdout:6: oid 6 version 5 is already newer than 4 2024-03-30T12:55:47.186 INFO:tasks.rados.rados.0.smithi067.stdout:6: oid 6 updating version 5 to 6 2024-03-30T12:55:47.186 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 6 v 6 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-03-30T12:55:47.186 INFO:tasks.rados.rados.0.smithi067.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-03-30T12:55:47.186 INFO:tasks.rados.rados.0.smithi067.stdout:7: finishing write tid 1 to smithi06731802-7 2024-03-30T12:55:47.187 INFO:tasks.rados.rados.0.smithi067.stdout:7: finishing write tid 2 to smithi06731802-7 2024-03-30T12:55:47.187 INFO:tasks.rados.rados.0.smithi067.stdout:8: finishing write tid 1 to smithi06731802-8 2024-03-30T12:55:47.187 INFO:tasks.rados.rados.0.smithi067.stdout:8: finishing write tid 2 to smithi06731802-8 2024-03-30T12:55:47.187 INFO:tasks.rados.rados.0.smithi067.stdout:8: finishing write tid 3 to smithi06731802-8 2024-03-30T12:55:47.187 INFO:tasks.rados.rados.0.smithi067.stdout:9: finishing write tid 1 to smithi06731802-9 2024-03-30T12:55:47.187 INFO:tasks.rados.rados.0.smithi067.stdout:8: finishing write tid 4 to smithi06731802-8 2024-03-30T12:55:47.187 INFO:tasks.rados.rados.0.smithi067.stdout:8: oid 8 updating version 0 to 3 2024-03-30T12:55:47.187 INFO:tasks.rados.rados.0.smithi067.stdout:8: oid 8 version 3 is already newer than 1 2024-03-30T12:55:47.187 INFO:tasks.rados.rados.0.smithi067.stdout:8: oid 8 version 3 is already newer than 2 2024-03-30T12:55:47.187 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-03-30T12:55:47.187 INFO:tasks.rados.rados.0.smithi067.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-03-30T12:55:47.187 INFO:tasks.rados.rados.0.smithi067.stdout:7: finishing write tid 3 to smithi06731802-7 2024-03-30T12:55:47.187 INFO:tasks.rados.rados.0.smithi067.stdout:7: finishing write tid 4 to smithi06731802-7 2024-03-30T12:55:47.187 INFO:tasks.rados.rados.0.smithi067.stdout:7: oid 7 updating version 0 to 3 2024-03-30T12:55:47.187 INFO:tasks.rados.rados.0.smithi067.stdout:7: oid 7 version 3 is already newer than 1 2024-03-30T12:55:47.188 INFO:tasks.rados.rados.0.smithi067.stdout:7: oid 7 version 3 is already newer than 2 2024-03-30T12:55:47.188 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-03-30T12:55:47.188 INFO:tasks.rados.rados.0.smithi067.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-03-30T12:55:47.188 INFO:tasks.rados.rados.0.smithi067.stdout:10: finishing write tid 1 to smithi06731802-10 2024-03-30T12:55:47.188 INFO:tasks.rados.rados.0.smithi067.stdout:10: finishing write tid 2 to smithi06731802-10 2024-03-30T12:55:47.188 INFO:tasks.rados.rados.0.smithi067.stdout:11: finishing write tid 1 to smithi06731802-11 2024-03-30T12:55:47.188 INFO:tasks.rados.rados.0.smithi067.stdout:11: finishing write tid 2 to smithi06731802-11 2024-03-30T12:55:47.188 INFO:tasks.rados.rados.0.smithi067.stdout:11: finishing write tid 3 to smithi06731802-11 2024-03-30T12:55:47.188 INFO:tasks.rados.rados.0.smithi067.stdout:11: finishing write tid 4 to smithi06731802-11 2024-03-30T12:55:47.188 INFO:tasks.rados.rados.0.smithi067.stdout:11: oid 11 updating version 0 to 3 2024-03-30T12:55:47.188 INFO:tasks.rados.rados.0.smithi067.stdout:11: oid 11 version 3 is already newer than 2 2024-03-30T12:55:47.188 INFO:tasks.rados.rados.0.smithi067.stdout:11: oid 11 version 3 is already newer than 1 2024-03-30T12:55:47.188 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-03-30T12:55:47.188 INFO:tasks.rados.rados.0.smithi067.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-03-30T12:55:47.188 INFO:tasks.rados.rados.0.smithi067.stdout:12: finishing write tid 1 to smithi06731802-12 2024-03-30T12:55:47.189 INFO:tasks.rados.rados.0.smithi067.stdout:12: finishing write tid 2 to smithi06731802-12 2024-03-30T12:55:47.189 INFO:tasks.rados.rados.0.smithi067.stdout:12: finishing write tid 3 to smithi06731802-12 2024-03-30T12:55:47.189 INFO:tasks.rados.rados.0.smithi067.stdout:12: finishing write tid 4 to smithi06731802-12 2024-03-30T12:55:47.189 INFO:tasks.rados.rados.0.smithi067.stdout:12: oid 12 updating version 0 to 4 2024-03-30T12:55:47.189 INFO:tasks.rados.rados.0.smithi067.stdout:12: oid 12 updating version 4 to 5 2024-03-30T12:55:47.189 INFO:tasks.rados.rados.0.smithi067.stdout:12: oid 12 updating version 5 to 6 2024-03-30T12:55:47.189 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 12 v 6 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-03-30T12:55:47.189 INFO:tasks.rados.rados.0.smithi067.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-03-30T12:55:47.189 INFO:tasks.rados.rados.0.smithi067.stdout:13: finishing write tid 1 to smithi06731802-13 2024-03-30T12:55:47.189 INFO:tasks.rados.rados.0.smithi067.stdout:13: finishing write tid 2 to smithi06731802-13 2024-03-30T12:55:47.189 INFO:tasks.rados.rados.0.smithi067.stdout:13: finishing write tid 3 to smithi06731802-13 2024-03-30T12:55:47.189 INFO:tasks.rados.rados.0.smithi067.stdout:2: done (15 left) 2024-03-30T12:55:47.189 INFO:tasks.rados.rados.0.smithi067.stdout:3: done (14 left) 2024-03-30T12:55:47.189 INFO:tasks.rados.rados.0.smithi067.stdout:4: done (13 left) 2024-03-30T12:55:47.189 INFO:tasks.rados.rados.0.smithi067.stdout:5: done (12 left) 2024-03-30T12:55:47.189 INFO:tasks.rados.rados.0.smithi067.stdout:6: done (11 left) 2024-03-30T12:55:47.190 INFO:tasks.rados.rados.0.smithi067.stdout:7: done (10 left) 2024-03-30T12:55:47.190 INFO:tasks.rados.rados.0.smithi067.stdout:8: done (9 left) 2024-03-30T12:55:47.190 INFO:tasks.rados.rados.0.smithi067.stdout:11: done (8 left) 2024-03-30T12:55:47.190 INFO:tasks.rados.rados.0.smithi067.stdout:12: done (7 left) 2024-03-30T12:55:47.190 INFO:tasks.rados.rados.0.smithi067.stdout:18: write initial oid 18 2024-03-30T12:55:47.190 INFO:tasks.rados.rados.0.smithi067.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2024-03-30T12:55:47.190 INFO:tasks.rados.rados.0.smithi067.stdout:18: writing smithi06731802-18 from 685719 to 1397658 tid 1 2024-03-30T12:55:47.191 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T12:55:47.193+0000 7fbc9da50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25191) UID: 0 2024-03-30T12:55:47.192 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:47.192 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:47.193 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-30T12:55:47.195 INFO:tasks.rados.rados.0.smithi067.stdout:18: writing smithi06731802-18 from 1845135 to 2285719 tid 2 2024-03-30T12:55:47.195 INFO:tasks.rados.rados.0.smithi067.stdout:19: write initial oid 19 2024-03-30T12:55:47.195 INFO:tasks.rados.rados.0.smithi067.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2024-03-30T12:55:47.204 INFO:tasks.rados.rados.0.smithi067.stdout:19: writing smithi06731802-19 from 702526 to 1489714 tid 1 2024-03-30T12:55:47.209 INFO:tasks.rados.rados.0.smithi067.stdout:19: writing smithi06731802-19 from 1903617 to 2302526 tid 2 2024-03-30T12:55:47.209 INFO:tasks.rados.rados.0.smithi067.stdout:20: write initial oid 20 2024-03-30T12:55:47.209 INFO:tasks.rados.rados.0.smithi067.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2024-03-30T12:55:47.210 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-03-30T12:55:47.215 INFO:tasks.rados.rados.0.smithi067.stdout:20: writing smithi06731802-20 from 719333 to 1181770 tid 1 2024-03-30T12:55:47.222 INFO:tasks.rados.rados.0.smithi067.stdout:20: writing smithi06731802-20 from 1962099 to 2319333 tid 2 2024-03-30T12:55:47.222 INFO:tasks.rados.rados.0.smithi067.stdout:21: write initial oid 21 2024-03-30T12:55:47.222 INFO:tasks.rados.rados.0.smithi067.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2024-03-30T12:55:47.229 INFO:tasks.rados.rados.0.smithi067.stdout:21: writing smithi06731802-21 from 736140 to 1273826 tid 1 2024-03-30T12:55:47.235 INFO:tasks.rados.rados.0.smithi067.stdout:21: writing smithi06731802-21 from 2020581 to 2336140 tid 2 2024-03-30T12:55:47.236 INFO:tasks.rados.rados.0.smithi067.stdout:22: write initial oid 22 2024-03-30T12:55:47.236 INFO:tasks.rados.rados.0.smithi067.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2024-03-30T12:55:47.244 INFO:tasks.rados.rados.0.smithi067.stdout:22: writing smithi06731802-22 from 752947 to 1365882 tid 1 2024-03-30T12:55:47.250 INFO:tasks.rados.rados.0.smithi067.stdout:22: writing smithi06731802-22 from 1962710 to 2352947 tid 2 2024-03-30T12:55:47.251 INFO:tasks.rados.rados.0.smithi067.stdout:23: write initial oid 23 2024-03-30T12:55:47.251 INFO:tasks.rados.rados.0.smithi067.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2024-03-30T12:55:47.258 INFO:tasks.rados.rados.0.smithi067.stdout:23: writing smithi06731802-23 from 769754 to 1457938 tid 1 2024-03-30T12:55:47.264 INFO:tasks.rados.rados.0.smithi067.stdout:23: writing smithi06731802-23 from 2021192 to 2369754 tid 2 2024-03-30T12:55:47.265 INFO:tasks.rados.rados.0.smithi067.stdout:24: write initial oid 24 2024-03-30T12:55:47.265 INFO:tasks.rados.rados.0.smithi067.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2024-03-30T12:55:47.272 INFO:tasks.rados.rados.0.smithi067.stdout:24: writing smithi06731802-24 from 786561 to 1266347 tid 1 2024-03-30T12:55:47.278 INFO:tasks.rados.rados.0.smithi067.stdout:24: writing smithi06731802-24 from 1796027 to 2263691 tid 2 2024-03-30T12:55:47.278 INFO:tasks.rados.rados.0.smithi067.stdout:25: write initial oid 25 2024-03-30T12:55:47.278 INFO:tasks.rados.rados.0.smithi067.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2024-03-30T12:55:47.284 INFO:tasks.rados.rados.0.smithi067.stdout:25: writing smithi06731802-25 from 403368 to 958403 tid 1 2024-03-30T12:55:47.291 INFO:tasks.rados.rados.0.smithi067.stdout:25: writing smithi06731802-25 from 1454509 to 2182184 tid 2 2024-03-30T12:55:47.292 INFO:tasks.rados.rados.0.smithi067.stdout:26: write initial oid 26 2024-03-30T12:55:47.292 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-03-30T12:55:47.293+0000 7f710ac50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25189) UID: 0 2024-03-30T12:55:47.292 INFO:tasks.rados.rados.0.smithi067.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2024-03-30T12:55:47.298 INFO:tasks.rados.rados.0.smithi067.stdout:26: writing smithi06731802-26 from 420175 to 1050459 tid 1 2024-03-30T12:55:47.307 INFO:tasks.rados.rados.0.smithi067.stdout:26: writing smithi06731802-26 from 1796638 to 2267971 tid 2 2024-03-30T12:55:47.308 INFO:tasks.rados.rados.0.smithi067.stdout: waiting on 16 2024-03-30T12:55:47.308 INFO:tasks.rados.rados.0.smithi067.stdout:9: finishing write tid 2 to smithi06731802-9 2024-03-30T12:55:47.308 INFO:tasks.rados.rados.0.smithi067.stdout:9: finishing write tid 3 to smithi06731802-9 2024-03-30T12:55:47.308 INFO:tasks.rados.rados.0.smithi067.stdout:14: finishing write tid 1 to smithi06731802-14 2024-03-30T12:55:47.308 INFO:tasks.rados.rados.0.smithi067.stdout:14: finishing write tid 2 to smithi06731802-14 2024-03-30T12:55:47.308 INFO:tasks.rados.rados.0.smithi067.stdout:13: finishing write tid 4 to smithi06731802-13 2024-03-30T12:55:47.308 INFO:tasks.rados.rados.0.smithi067.stdout:13: oid 13 updating version 0 to 5 2024-03-30T12:55:47.308 INFO:tasks.rados.rados.0.smithi067.stdout:13: oid 13 updating version 5 to 6 2024-03-30T12:55:47.308 INFO:tasks.rados.rados.0.smithi067.stdout:13: oid 13 version 6 is already newer than 4 2024-03-30T12:55:47.308 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-03-30T12:55:47.308 INFO:tasks.rados.rados.0.smithi067.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-03-30T12:55:47.308 INFO:tasks.rados.rados.0.smithi067.stdout:15: finishing write tid 1 to smithi06731802-15 2024-03-30T12:55:47.308 INFO:tasks.rados.rados.0.smithi067.stdout:15: finishing write tid 2 to smithi06731802-15 2024-03-30T12:55:47.308 INFO:tasks.rados.rados.0.smithi067.stdout:15: finishing write tid 3 to smithi06731802-15 2024-03-30T12:55:47.308 INFO:tasks.rados.rados.0.smithi067.stdout:14: finishing write tid 3 to smithi06731802-14 2024-03-30T12:55:47.309 INFO:tasks.rados.rados.0.smithi067.stdout:9: finishing write tid 4 to smithi06731802-9 2024-03-30T12:55:47.309 INFO:tasks.rados.rados.0.smithi067.stdout:9: oid 9 updating version 0 to 6 2024-03-30T12:55:47.309 INFO:tasks.rados.rados.0.smithi067.stdout:9: oid 9 version 6 is already newer than 5 2024-03-30T12:55:47.309 INFO:tasks.rados.rados.0.smithi067.stdout:9: oid 9 version 6 is already newer than 4 2024-03-30T12:55:47.309 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 9 v 6 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-03-30T12:55:47.309 INFO:tasks.rados.rados.0.smithi067.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-03-30T12:55:47.309 INFO:tasks.rados.rados.0.smithi067.stdout:16: finishing write tid 1 to smithi06731802-16 2024-03-30T12:55:47.309 INFO:tasks.rados.rados.0.smithi067.stdout:14: finishing write tid 4 to smithi06731802-14 2024-03-30T12:55:47.309 INFO:tasks.rados.rados.0.smithi067.stdout:14: oid 14 updating version 0 to 9 2024-03-30T12:55:47.309 INFO:tasks.rados.rados.0.smithi067.stdout:14: oid 14 version 9 is already newer than 8 2024-03-30T12:55:47.309 INFO:tasks.rados.rados.0.smithi067.stdout:14: oid 14 version 9 is already newer than 7 2024-03-30T12:55:47.309 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 14 v 9 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-03-30T12:55:47.309 INFO:tasks.rados.rados.0.smithi067.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-03-30T12:55:47.309 INFO:tasks.rados.rados.0.smithi067.stdout:17: finishing write tid 1 to smithi06731802-17 2024-03-30T12:55:47.310 INFO:tasks.rados.rados.0.smithi067.stdout:17: finishing write tid 2 to smithi06731802-17 2024-03-30T12:55:47.310 INFO:tasks.rados.rados.0.smithi067.stdout:17: finishing write tid 3 to smithi06731802-17 2024-03-30T12:55:47.310 INFO:tasks.rados.rados.0.smithi067.stdout:17: finishing write tid 4 to smithi06731802-17 2024-03-30T12:55:47.310 INFO:tasks.rados.rados.0.smithi067.stdout:17: oid 17 updating version 0 to 3 2024-03-30T12:55:47.310 INFO:tasks.rados.rados.0.smithi067.stdout:17: oid 17 version 3 is already newer than 2 2024-03-30T12:55:47.310 INFO:tasks.rados.rados.0.smithi067.stdout:17: oid 17 version 3 is already newer than 1 2024-03-30T12:55:47.310 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 17 v 3 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-03-30T12:55:47.310 INFO:tasks.rados.rados.0.smithi067.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-03-30T12:55:47.310 INFO:tasks.rados.rados.0.smithi067.stdout:10: finishing write tid 3 to smithi06731802-10 2024-03-30T12:55:47.310 INFO:tasks.rados.rados.0.smithi067.stdout:9: done (15 left) 2024-03-30T12:55:47.310 INFO:tasks.rados.rados.0.smithi067.stdout:13: done (14 left) 2024-03-30T12:55:47.310 INFO:tasks.rados.rados.0.smithi067.stdout:14: done (13 left) 2024-03-30T12:55:47.310 INFO:tasks.rados.rados.0.smithi067.stdout:17: done (12 left) 2024-03-30T12:55:47.310 INFO:tasks.rados.rados.0.smithi067.stdout:27: write initial oid 27 2024-03-30T12:55:47.311 INFO:tasks.rados.rados.0.smithi067.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2024-03-30T12:55:47.312 INFO:tasks.rados.rados.0.smithi067.stdout:27: writing smithi06731802-27 from 436982 to 1142515 tid 1 2024-03-30T12:55:47.320 INFO:tasks.rados.rados.0.smithi067.stdout:27: writing smithi06731802-27 from 1855120 to 2436982 tid 2 2024-03-30T12:55:47.320 INFO:tasks.rados.rados.0.smithi067.stdout:28: write initial oid 28 2024-03-30T12:55:47.320 INFO:tasks.rados.rados.0.smithi067.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2024-03-30T12:55:47.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:47.321+0000 7ffa3ba69640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:47.327 INFO:tasks.rados.rados.0.smithi067.stdout:28: writing smithi06731802-28 from 453789 to 1234571 tid 1 2024-03-30T12:55:47.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:47.329+0000 7ffa3ba69640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:47.334 INFO:tasks.rados.rados.0.smithi067.stdout:28: writing smithi06731802-28 from 1913602 to 2388604 tid 2 2024-03-30T12:55:47.334 INFO:tasks.rados.rados.0.smithi067.stdout:29: write initial oid 29 2024-03-30T12:55:47.334 INFO:tasks.rados.rados.0.smithi067.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2024-03-30T12:55:47.339 INFO:tasks.rados.rados.0.smithi067.stdout:29: writing smithi06731802-29 from 470596 to 926627 tid 1 2024-03-30T12:55:47.347 INFO:tasks.rados.rados.0.smithi067.stdout:29: writing smithi06731802-29 from 1572084 to 2190744 tid 2 2024-03-30T12:55:47.348 INFO:tasks.rados.rados.0.smithi067.stdout:30: write initial oid 30 2024-03-30T12:55:47.348 INFO:tasks.rados.rados.0.smithi067.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2024-03-30T12:55:47.354 INFO:tasks.rados.rados.0.smithi067.stdout:30: writing smithi06731802-30 from 487403 to 1018683 tid 1 2024-03-30T12:55:47.360 INFO:tasks.rados.rados.0.smithi067.stdout:30: writing smithi06731802-30 from 1514213 to 1992884 tid 2 2024-03-30T12:55:47.360 INFO:tasks.rados.rados.0.smithi067.stdout: waiting on 16 2024-03-30T12:55:47.360 INFO:tasks.rados.rados.0.smithi067.stdout:18: finishing write tid 1 to smithi06731802-18 2024-03-30T12:55:47.360 INFO:tasks.rados.rados.0.smithi067.stdout:10: finishing write tid 4 to smithi06731802-10 2024-03-30T12:55:47.360 INFO:tasks.rados.rados.0.smithi067.stdout:10: oid 10 updating version 0 to 9 2024-03-30T12:55:47.360 INFO:tasks.rados.rados.0.smithi067.stdout:10: oid 10 version 9 is already newer than 7 2024-03-30T12:55:47.360 INFO:tasks.rados.rados.0.smithi067.stdout:10: oid 10 version 9 is already newer than 8 2024-03-30T12:55:47.360 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 10 v 9 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-03-30T12:55:47.360 INFO:tasks.rados.rados.0.smithi067.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-03-30T12:55:47.361 INFO:tasks.rados.rados.0.smithi067.stdout:18: finishing write tid 2 to smithi06731802-18 2024-03-30T12:55:47.361 INFO:tasks.rados.rados.0.smithi067.stdout:18: finishing write tid 3 to smithi06731802-18 2024-03-30T12:55:47.361 INFO:tasks.rados.rados.0.smithi067.stdout:15: finishing write tid 4 to smithi06731802-15 2024-03-30T12:55:47.361 INFO:tasks.rados.rados.0.smithi067.stdout:15: oid 15 updating version 0 to 1 2024-03-30T12:55:47.361 INFO:tasks.rados.rados.0.smithi067.stdout:15: oid 15 updating version 1 to 3 2024-03-30T12:55:47.361 INFO:tasks.rados.rados.0.smithi067.stdout:15: oid 15 version 3 is already newer than 2 2024-03-30T12:55:47.361 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 15 v 3 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-03-30T12:55:47.361 INFO:tasks.rados.rados.0.smithi067.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-03-30T12:55:47.361 INFO:tasks.rados.rados.0.smithi067.stdout:18: finishing write tid 4 to smithi06731802-18 2024-03-30T12:55:47.361 INFO:tasks.rados.rados.0.smithi067.stdout:18: oid 18 updating version 0 to 2 2024-03-30T12:55:47.361 INFO:tasks.rados.rados.0.smithi067.stdout:18: oid 18 version 2 is already newer than 1 2024-03-30T12:55:47.361 INFO:tasks.rados.rados.0.smithi067.stdout:18: oid 18 updating version 2 to 3 2024-03-30T12:55:47.361 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 18 v 3 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-03-30T12:55:47.361 INFO:tasks.rados.rados.0.smithi067.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-03-30T12:55:47.361 INFO:tasks.rados.rados.0.smithi067.stdout:19: finishing write tid 1 to smithi06731802-19 2024-03-30T12:55:47.362 INFO:tasks.rados.rados.0.smithi067.stdout:10: done (15 left) 2024-03-30T12:55:47.362 INFO:tasks.rados.rados.0.smithi067.stdout:15: done (14 left) 2024-03-30T12:55:47.362 INFO:tasks.rados.rados.0.smithi067.stdout:18: done (13 left) 2024-03-30T12:55:47.362 INFO:tasks.rados.rados.0.smithi067.stdout:31: write initial oid 31 2024-03-30T12:55:47.362 INFO:tasks.rados.rados.0.smithi067.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2024-03-30T12:55:47.365 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0: {} 2024-03-30T12:55:47.365 INFO:teuthology.orchestra.run.smithi067.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-30T12:55:47.366 INFO:tasks.rados.rados.0.smithi067.stdout:31: writing smithi06731802-31 from 504210 to 1110739 tid 1 2024-03-30T12:55:47.372 INFO:tasks.rados.rados.0.smithi067.stdout:31: writing smithi06731802-31 from 1572695 to 2195024 tid 2 2024-03-30T12:55:47.372 INFO:tasks.rados.rados.0.smithi067.stdout:32: write initial oid 32 2024-03-30T12:55:47.372 INFO:tasks.rados.rados.0.smithi067.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2024-03-30T12:55:47.380 INFO:tasks.rados.rados.0.smithi067.stdout:32: writing smithi06731802-32 from 521017 to 1319148 tid 1 2024-03-30T12:55:47.386 INFO:tasks.rados.rados.0.smithi067.stdout:32: writing smithi06731802-32 from 1747530 to 2229870 tid 2 2024-03-30T12:55:47.386 INFO:tasks.rados.rados.0.smithi067.stdout:33: write initial oid 33 2024-03-30T12:55:47.386 INFO:tasks.rados.rados.0.smithi067.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2024-03-30T12:55:47.387 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1: {} 2024-03-30T12:55:47.387 INFO:teuthology.orchestra.run.smithi067.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-30T12:55:47.392 INFO:tasks.rados.rados.0.smithi067.stdout:33: writing smithi06731802-33 from 537824 to 1011204 tid 1 2024-03-30T12:55:47.393 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T12:55:47.391+0000 7f75d16be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25273) UID: 0 2024-03-30T12:55:47.396 INFO:teuthology.orchestra.run.smithi082.stdout:ERROR: (22) Invalid argument 2024-03-30T12:55:47.397 INFO:teuthology.orchestra.run.smithi082.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-30T12:55:47.400 INFO:tasks.rados.rados.0.smithi067.stdout:33: writing smithi06731802-33 from 1806012 to 2432010 tid 2 2024-03-30T12:55:47.400 INFO:tasks.rados.rados.0.smithi067.stdout: waiting on 16 2024-03-30T12:55:47.401 INFO:tasks.rados.rados.0.smithi067.stdout:19: finishing write tid 2 to smithi06731802-19 2024-03-30T12:55:47.401 INFO:tasks.rados.rados.0.smithi067.stdout:20: finishing write tid 1 to smithi06731802-20 2024-03-30T12:55:47.401 INFO:tasks.rados.rados.0.smithi067.stdout:20: finishing write tid 2 to smithi06731802-20 2024-03-30T12:55:47.401 INFO:tasks.rados.rados.0.smithi067.stdout:20: finishing write tid 3 to smithi06731802-20 2024-03-30T12:55:47.401 INFO:tasks.rados.rados.0.smithi067.stdout:20: finishing write tid 4 to smithi06731802-20 2024-03-30T12:55:47.401 INFO:tasks.rados.rados.0.smithi067.stdout:20: oid 20 updating version 0 to 4 2024-03-30T12:55:47.401 INFO:tasks.rados.rados.0.smithi067.stdout:20: oid 20 updating version 4 to 5 2024-03-30T12:55:47.402 INFO:tasks.rados.rados.0.smithi067.stdout:20: oid 20 updating version 5 to 6 2024-03-30T12:55:47.402 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 20 v 6 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-03-30T12:55:47.402 INFO:tasks.rados.rados.0.smithi067.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-03-30T12:55:47.402 INFO:tasks.rados.rados.0.smithi067.stdout:19: finishing write tid 3 to smithi06731802-19 2024-03-30T12:55:47.402 INFO:tasks.rados.rados.0.smithi067.stdout:19: finishing write tid 4 to smithi06731802-19 2024-03-30T12:55:47.402 INFO:tasks.rados.rados.0.smithi067.stdout:19: oid 19 updating version 0 to 5 2024-03-30T12:55:47.402 INFO:tasks.rados.rados.0.smithi067.stdout:19: oid 19 version 5 is already newer than 4 2024-03-30T12:55:47.402 INFO:tasks.rados.rados.0.smithi067.stdout:19: oid 19 updating version 5 to 6 2024-03-30T12:55:47.402 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 19 v 6 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-03-30T12:55:47.402 INFO:tasks.rados.rados.0.smithi067.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-03-30T12:55:47.402 INFO:tasks.rados.rados.0.smithi067.stdout:21: finishing write tid 1 to smithi06731802-21 2024-03-30T12:55:47.402 INFO:tasks.rados.rados.0.smithi067.stdout:21: finishing write tid 2 to smithi06731802-21 2024-03-30T12:55:47.402 INFO:tasks.rados.rados.0.smithi067.stdout:19: done (15 left) 2024-03-30T12:55:47.403 INFO:tasks.rados.rados.0.smithi067.stdout:20: done (14 left) 2024-03-30T12:55:47.403 INFO:tasks.rados.rados.0.smithi067.stdout:34: write initial oid 34 2024-03-30T12:55:47.403 INFO:tasks.rados.rados.0.smithi067.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2024-03-30T12:55:47.406 INFO:tasks.rados.rados.0.smithi067.stdout:34: writing smithi06731802-34 from 554631 to 1103260 tid 1 2024-03-30T12:55:47.409 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2: {} 2024-03-30T12:55:47.409 INFO:teuthology.orchestra.run.smithi067.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-30T12:55:47.413 INFO:tasks.rados.rados.0.smithi067.stdout:34: writing smithi06731802-34 from 1748141 to 2234150 tid 2 2024-03-30T12:55:47.413 INFO:tasks.rados.rados.0.smithi067.stdout:35: write initial oid 35 2024-03-30T12:55:47.413 INFO:tasks.rados.rados.0.smithi067.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2024-03-30T12:55:47.414 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-30T12:55:47.420 INFO:tasks.rados.rados.0.smithi067.stdout:35: writing smithi06731802-35 from 571438 to 1195316 tid 1 2024-03-30T12:55:47.428 INFO:tasks.rados.rados.0.smithi067.stdout:35: writing smithi06731802-35 from 1806623 to 2436290 tid 2 2024-03-30T12:55:47.428 INFO:tasks.rados.rados.0.smithi067.stdout: waiting on 16 2024-03-30T12:55:47.428 INFO:tasks.rados.rados.0.smithi067.stdout:21: finishing write tid 3 to smithi06731802-21 2024-03-30T12:55:47.428 INFO:tasks.rados.rados.0.smithi067.stdout:22: finishing write tid 1 to smithi06731802-22 2024-03-30T12:55:47.428 INFO:tasks.rados.rados.0.smithi067.stdout:21: finishing write tid 4 to smithi06731802-21 2024-03-30T12:55:47.428 INFO:tasks.rados.rados.0.smithi067.stdout:21: oid 21 updating version 0 to 1 2024-03-30T12:55:47.428 INFO:tasks.rados.rados.0.smithi067.stdout:21: oid 21 updating version 1 to 3 2024-03-30T12:55:47.428 INFO:tasks.rados.rados.0.smithi067.stdout:21: oid 21 version 3 is already newer than 2 2024-03-30T12:55:47.428 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 21 v 3 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-03-30T12:55:47.428 INFO:tasks.rados.rados.0.smithi067.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-03-30T12:55:47.428 INFO:tasks.rados.rados.0.smithi067.stdout:22: finishing write tid 2 to smithi06731802-22 2024-03-30T12:55:47.429 INFO:tasks.rados.rados.0.smithi067.stdout:22: finishing write tid 3 to smithi06731802-22 2024-03-30T12:55:47.429 INFO:tasks.rados.rados.0.smithi067.stdout:21: done (15 left) 2024-03-30T12:55:47.429 INFO:tasks.rados.rados.0.smithi067.stdout:36: write initial oid 36 2024-03-30T12:55:47.429 INFO:tasks.rados.rados.0.smithi067.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2024-03-30T12:55:47.430 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3: {} 2024-03-30T12:55:47.431 INFO:teuthology.orchestra.run.smithi067.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-30T12:55:47.435 INFO:tasks.rados.rados.0.smithi067.stdout:36: writing smithi06731802-36 from 588245 to 1287372 tid 1 2024-03-30T12:55:47.441 INFO:tasks.rados.rados.0.smithi067.stdout:36: writing smithi06731802-36 from 1865105 to 2354783 tid 2 2024-03-30T12:55:47.441 INFO:tasks.rados.rados.0.smithi067.stdout: waiting on 16 2024-03-30T12:55:47.441 INFO:tasks.rados.rados.0.smithi067.stdout:22: finishing write tid 4 to smithi06731802-22 2024-03-30T12:55:47.442 INFO:tasks.rados.rados.0.smithi067.stdout:22: oid 22 updating version 0 to 9 2024-03-30T12:55:47.442 INFO:tasks.rados.rados.0.smithi067.stdout:22: oid 22 version 9 is already newer than 8 2024-03-30T12:55:47.442 INFO:tasks.rados.rados.0.smithi067.stdout:22: oid 22 version 9 is already newer than 7 2024-03-30T12:55:47.442 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 22 v 9 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-03-30T12:55:47.442 INFO:tasks.rados.rados.0.smithi067.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-03-30T12:55:47.442 INFO:tasks.rados.rados.0.smithi067.stdout:16: finishing write tid 2 to smithi06731802-16 2024-03-30T12:55:47.442 INFO:tasks.rados.rados.0.smithi067.stdout:22: done (15 left) 2024-03-30T12:55:47.442 INFO:tasks.rados.rados.0.smithi067.stdout:37: write initial oid 37 2024-03-30T12:55:47.442 INFO:tasks.rados.rados.0.smithi067.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2024-03-30T12:55:47.449 INFO:tasks.rados.rados.0.smithi067.stdout:37: writing smithi06731802-37 from 605052 to 1379428 tid 1 2024-03-30T12:55:47.453 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4: {} 2024-03-30T12:55:47.454 INFO:teuthology.orchestra.run.smithi067.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-30T12:55:47.456 INFO:tasks.rados.rados.0.smithi067.stdout:37: writing smithi06731802-37 from 1923587 to 2556923 tid 2 2024-03-30T12:55:47.457 INFO:tasks.rados.rados.0.smithi067.stdout: waiting on 16 2024-03-30T12:55:47.457 INFO:tasks.rados.rados.0.smithi067.stdout:23: finishing write tid 1 to smithi06731802-23 2024-03-30T12:55:47.457 INFO:tasks.rados.rados.0.smithi067.stdout:23: finishing write tid 2 to smithi06731802-23 2024-03-30T12:55:47.457 INFO:tasks.rados.rados.0.smithi067.stdout:23: finishing write tid 3 to smithi06731802-23 2024-03-30T12:55:47.457 INFO:tasks.rados.rados.0.smithi067.stdout:24: finishing write tid 1 to smithi06731802-24 2024-03-30T12:55:47.457 INFO:tasks.rados.rados.0.smithi067.stdout:23: finishing write tid 4 to smithi06731802-23 2024-03-30T12:55:47.457 INFO:tasks.rados.rados.0.smithi067.stdout:23: oid 23 updating version 0 to 9 2024-03-30T12:55:47.457 INFO:tasks.rados.rados.0.smithi067.stdout:23: oid 23 version 9 is already newer than 7 2024-03-30T12:55:47.457 INFO:tasks.rados.rados.0.smithi067.stdout:23: oid 23 version 9 is already newer than 8 2024-03-30T12:55:47.457 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 23 v 9 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-03-30T12:55:47.457 INFO:tasks.rados.rados.0.smithi067.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-03-30T12:55:47.457 INFO:tasks.rados.rados.0.smithi067.stdout:24: finishing write tid 2 to smithi06731802-24 2024-03-30T12:55:47.457 INFO:tasks.rados.rados.0.smithi067.stdout:24: finishing write tid 3 to smithi06731802-24 2024-03-30T12:55:47.457 INFO:tasks.rados.rados.0.smithi067.stdout:24: finishing write tid 4 to smithi06731802-24 2024-03-30T12:55:47.457 INFO:tasks.rados.rados.0.smithi067.stdout:24: oid 24 updating version 0 to 10 2024-03-30T12:55:47.458 INFO:tasks.rados.rados.0.smithi067.stdout:24: oid 24 updating version 10 to 11 2024-03-30T12:55:47.458 INFO:tasks.rados.rados.0.smithi067.stdout:24: oid 24 updating version 11 to 12 2024-03-30T12:55:47.458 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 24 v 12 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-03-30T12:55:47.458 INFO:tasks.rados.rados.0.smithi067.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-03-30T12:55:47.458 INFO:tasks.rados.rados.0.smithi067.stdout:25: finishing write tid 1 to smithi06731802-25 2024-03-30T12:55:47.458 INFO:tasks.rados.rados.0.smithi067.stdout:23: done (15 left) 2024-03-30T12:55:47.458 INFO:tasks.rados.rados.0.smithi067.stdout:24: done (14 left) 2024-03-30T12:55:47.458 INFO:tasks.rados.rados.0.smithi067.stdout:38: write initial oid 38 2024-03-30T12:55:47.458 INFO:tasks.rados.rados.0.smithi067.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2024-03-30T12:55:47.462 INFO:tasks.rados.rados.0.smithi067.stdout:38: writing smithi06731802-38 from 621859 to 1071484 tid 1 2024-03-30T12:55:47.471 INFO:tasks.rados.rados.0.smithi067.stdout:38: writing smithi06731802-38 from 1865716 to 2621859 tid 2 2024-03-30T12:55:47.472 INFO:tasks.rados.rados.0.smithi067.stdout:39: write initial oid 39 2024-03-30T12:55:47.472 INFO:tasks.rados.rados.0.smithi067.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2024-03-30T12:55:47.478 INFO:tasks.rados.rados.0.smithi067.stdout:39: writing smithi06731802-39 from 638666 to 1163540 tid 1 2024-03-30T12:55:47.481 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5: {} 2024-03-30T12:55:47.482 INFO:teuthology.orchestra.run.smithi067.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-30T12:55:47.487 INFO:tasks.rados.rados.0.smithi067.stdout:39: writing smithi06731802-39 from 1924198 to 2561203 tid 2 2024-03-30T12:55:47.487 INFO:tasks.rados.rados.0.smithi067.stdout: waiting on 16 2024-03-30T12:55:47.487 INFO:tasks.rados.rados.0.smithi067.stdout:25: finishing write tid 2 to smithi06731802-25 2024-03-30T12:55:47.488 INFO:tasks.rados.rados.0.smithi067.stdout:25: finishing write tid 3 to smithi06731802-25 2024-03-30T12:55:47.488 INFO:tasks.rados.rados.0.smithi067.stdout:25: finishing write tid 4 to smithi06731802-25 2024-03-30T12:55:47.488 INFO:tasks.rados.rados.0.smithi067.stdout:25: oid 25 updating version 0 to 7 2024-03-30T12:55:47.488 INFO:tasks.rados.rados.0.smithi067.stdout:25: oid 25 updating version 7 to 8 2024-03-30T12:55:47.488 INFO:tasks.rados.rados.0.smithi067.stdout:25: oid 25 updating version 8 to 9 2024-03-30T12:55:47.488 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 25 v 9 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-03-30T12:55:47.488 INFO:tasks.rados.rados.0.smithi067.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-03-30T12:55:47.489 INFO:tasks.rados.rados.0.smithi067.stdout:26: finishing write tid 1 to smithi06731802-26 2024-03-30T12:55:47.489 INFO:tasks.rados.rados.0.smithi067.stdout:26: finishing write tid 2 to smithi06731802-26 2024-03-30T12:55:47.489 INFO:tasks.rados.rados.0.smithi067.stdout:26: finishing write tid 3 to smithi06731802-26 2024-03-30T12:55:47.489 INFO:tasks.rados.rados.0.smithi067.stdout:26: finishing write tid 4 to smithi06731802-26 2024-03-30T12:55:47.489 INFO:tasks.rados.rados.0.smithi067.stdout:26: oid 26 updating version 0 to 5 2024-03-30T12:55:47.489 INFO:tasks.rados.rados.0.smithi067.stdout:26: oid 26 version 5 is already newer than 4 2024-03-30T12:55:47.489 INFO:tasks.rados.rados.0.smithi067.stdout:26: oid 26 updating version 5 to 6 2024-03-30T12:55:47.489 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-03-30T12:55:47.489 INFO:tasks.rados.rados.0.smithi067.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-03-30T12:55:47.489 INFO:tasks.rados.rados.0.smithi067.stdout:27: finishing write tid 1 to smithi06731802-27 2024-03-30T12:55:47.489 INFO:tasks.rados.rados.0.smithi067.stdout:27: finishing write tid 2 to smithi06731802-27 2024-03-30T12:55:47.490 INFO:tasks.rados.rados.0.smithi067.stdout:27: finishing write tid 3 to smithi06731802-27 2024-03-30T12:55:47.490 INFO:tasks.rados.rados.0.smithi067.stdout:28: finishing write tid 1 to smithi06731802-28 2024-03-30T12:55:47.490 INFO:tasks.rados.rados.0.smithi067.stdout:25: done (15 left) 2024-03-30T12:55:47.490 INFO:tasks.rados.rados.0.smithi067.stdout:26: done (14 left) 2024-03-30T12:55:47.490 INFO:tasks.rados.rados.0.smithi067.stdout:40: write initial oid 40 2024-03-30T12:55:47.490 INFO:tasks.rados.rados.0.smithi067.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2024-03-30T12:55:47.493 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T12:55:47.493+0000 7fbc9da50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25191) UID: 0 2024-03-30T12:55:47.495 INFO:tasks.rados.rados.0.smithi067.stdout:40: writing smithi06731802-40 from 655473 to 1371949 tid 1 2024-03-30T12:55:47.502 INFO:tasks.rados.rados.0.smithi067.stdout:40: writing smithi06731802-40 from 2099033 to 2655473 tid 2 2024-03-30T12:55:47.503 INFO:tasks.rados.rados.0.smithi067.stdout:41: write initial oid 41 2024-03-30T12:55:47.503 INFO:tasks.rados.rados.0.smithi067.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2024-03-30T12:55:47.507 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6: {} 2024-03-30T12:55:47.507 INFO:teuthology.orchestra.run.smithi067.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-30T12:55:47.511 INFO:tasks.rados.rados.0.smithi067.stdout:41: writing smithi06731802-41 from 672280 to 1464005 tid 1 2024-03-30T12:55:47.519 INFO:tasks.rados.rados.0.smithi067.stdout:41: writing smithi06731802-41 from 2157515 to 2672280 tid 2 2024-03-30T12:55:47.519 INFO:tasks.rados.rados.0.smithi067.stdout: waiting on 16 2024-03-30T12:55:47.520 INFO:tasks.rados.rados.0.smithi067.stdout:28: finishing write tid 2 to smithi06731802-28 2024-03-30T12:55:47.520 INFO:tasks.rados.rados.0.smithi067.stdout:28: finishing write tid 3 to smithi06731802-28 2024-03-30T12:55:47.520 INFO:tasks.rados.rados.0.smithi067.stdout:28: finishing write tid 4 to smithi06731802-28 2024-03-30T12:55:47.520 INFO:tasks.rados.rados.0.smithi067.stdout:28: oid 28 updating version 0 to 10 2024-03-30T12:55:47.520 INFO:tasks.rados.rados.0.smithi067.stdout:28: oid 28 updating version 10 to 12 2024-03-30T12:55:47.520 INFO:tasks.rados.rados.0.smithi067.stdout:28: oid 28 version 12 is already newer than 11 2024-03-30T12:55:47.521 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 28 v 12 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-03-30T12:55:47.521 INFO:tasks.rados.rados.0.smithi067.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-03-30T12:55:47.521 INFO:tasks.rados.rados.0.smithi067.stdout:29: finishing write tid 1 to smithi06731802-29 2024-03-30T12:55:47.521 INFO:tasks.rados.rados.0.smithi067.stdout:29: finishing write tid 2 to smithi06731802-29 2024-03-30T12:55:47.521 INFO:tasks.rados.rados.0.smithi067.stdout:29: finishing write tid 3 to smithi06731802-29 2024-03-30T12:55:47.521 INFO:tasks.rados.rados.0.smithi067.stdout:29: finishing write tid 4 to smithi06731802-29 2024-03-30T12:55:47.522 INFO:tasks.rados.rados.0.smithi067.stdout:29: oid 29 updating version 0 to 1 2024-03-30T12:55:47.522 INFO:tasks.rados.rados.0.smithi067.stdout:29: oid 29 updating version 1 to 2 2024-03-30T12:55:47.522 INFO:tasks.rados.rados.0.smithi067.stdout:29: oid 29 updating version 2 to 3 2024-03-30T12:55:47.522 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 29 v 3 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-03-30T12:55:47.522 INFO:tasks.rados.rados.0.smithi067.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-03-30T12:55:47.522 INFO:tasks.rados.rados.0.smithi067.stdout:30: finishing write tid 1 to smithi06731802-30 2024-03-30T12:55:47.522 INFO:tasks.rados.rados.0.smithi067.stdout:30: finishing write tid 2 to smithi06731802-30 2024-03-30T12:55:47.523 INFO:tasks.rados.rados.0.smithi067.stdout:28: done (15 left) 2024-03-30T12:55:47.523 INFO:tasks.rados.rados.0.smithi067.stdout:29: done (14 left) 2024-03-30T12:55:47.523 INFO:tasks.rados.rados.0.smithi067.stdout:42: write initial oid 42 2024-03-30T12:55:47.523 INFO:tasks.rados.rados.0.smithi067.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2024-03-30T12:55:47.525 INFO:tasks.rados.rados.0.smithi067.stdout:42: writing smithi06731802-42 from 689087 to 1156061 tid 1 2024-03-30T12:55:47.533 INFO:tasks.rados.rados.0.smithi067.stdout:42: writing smithi06731802-42 from 1699644 to 2483976 tid 2 2024-03-30T12:55:47.533 INFO:tasks.rados.rados.0.smithi067.stdout:43: write initial oid 43 2024-03-30T12:55:47.533 INFO:tasks.rados.rados.0.smithi067.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2024-03-30T12:55:47.534 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7: {} 2024-03-30T12:55:47.534 INFO:teuthology.orchestra.run.smithi067.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-30T12:55:47.540 INFO:tasks.rados.rados.0.smithi067.stdout:43: writing smithi06731802-43 from 705894 to 1248117 tid 1 2024-03-30T12:55:47.548 INFO:tasks.rados.rados.0.smithi067.stdout:43: writing smithi06731802-43 from 1758126 to 2402469 tid 2 2024-03-30T12:55:47.548 INFO:tasks.rados.rados.0.smithi067.stdout: waiting on 16 2024-03-30T12:55:47.548 INFO:tasks.rados.rados.0.smithi067.stdout:30: finishing write tid 3 to smithi06731802-30 2024-03-30T12:55:47.548 INFO:tasks.rados.rados.0.smithi067.stdout:30: finishing write tid 4 to smithi06731802-30 2024-03-30T12:55:47.548 INFO:tasks.rados.rados.0.smithi067.stdout:30: oid 30 updating version 0 to 2 2024-03-30T12:55:47.548 INFO:tasks.rados.rados.0.smithi067.stdout:30: oid 30 updating version 2 to 3 2024-03-30T12:55:47.548 INFO:tasks.rados.rados.0.smithi067.stdout:30: oid 30 version 3 is already newer than 1 2024-03-30T12:55:47.548 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 30 v 3 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-03-30T12:55:47.548 INFO:tasks.rados.rados.0.smithi067.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-03-30T12:55:47.548 INFO:tasks.rados.rados.0.smithi067.stdout:16: finishing write tid 3 to smithi06731802-16 2024-03-30T12:55:47.548 INFO:tasks.rados.rados.0.smithi067.stdout:30: done (15 left) 2024-03-30T12:55:47.548 INFO:tasks.rados.rados.0.smithi067.stdout:44: write initial oid 44 2024-03-30T12:55:47.549 INFO:tasks.rados.rados.0.smithi067.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2024-03-30T12:55:47.555 INFO:tasks.rados.rados.0.smithi067.stdout:44: writing smithi06731802-44 from 722701 to 1340173 tid 1 2024-03-30T12:55:47.563 INFO:tasks.rados.rados.0.smithi067.stdout:44: writing smithi06731802-44 from 1816608 to 2604609 tid 2 2024-03-30T12:55:47.563 INFO:tasks.rados.rados.0.smithi067.stdout: waiting on 16 2024-03-30T12:55:47.563 INFO:tasks.rados.rados.0.smithi067.stdout:16: finishing write tid 4 to smithi06731802-16 2024-03-30T12:55:47.563 INFO:tasks.rados.rados.0.smithi067.stdout:16: oid 16 updating version 0 to 12 2024-03-30T12:55:47.564 INFO:tasks.rados.rados.0.smithi067.stdout:16: oid 16 version 12 is already newer than 11 2024-03-30T12:55:47.564 INFO:tasks.rados.rados.0.smithi067.stdout:16: oid 16 version 12 is already newer than 10 2024-03-30T12:55:47.564 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 16 v 12 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-03-30T12:55:47.564 INFO:tasks.rados.rados.0.smithi067.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-03-30T12:55:47.564 INFO:tasks.rados.rados.0.smithi067.stdout:31: finishing write tid 1 to smithi06731802-31 2024-03-30T12:55:47.564 INFO:tasks.rados.rados.0.smithi067.stdout:31: finishing write tid 2 to smithi06731802-31 2024-03-30T12:55:47.564 INFO:tasks.rados.rados.0.smithi067.stdout:16: done (15 left) 2024-03-30T12:55:47.564 INFO:tasks.rados.rados.0.smithi067.stdout:45: write initial oid 45 2024-03-30T12:55:47.564 INFO:tasks.rados.rados.0.smithi067.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2024-03-30T12:55:47.571 INFO:tasks.rados.rados.0.smithi067.stdout:45: writing smithi06731802-45 from 739508 to 1432229 tid 1 2024-03-30T12:55:47.578 INFO:tasks.rados.rados.0.smithi067.stdout:45: writing smithi06731802-45 from 1875090 to 2523102 tid 2 2024-03-30T12:55:47.578 INFO:tasks.rados.rados.0.smithi067.stdout: waiting on 16 2024-03-30T12:55:47.578 INFO:tasks.rados.rados.0.smithi067.stdout:32: finishing write tid 1 to smithi06731802-32 2024-03-30T12:55:47.579 INFO:tasks.rados.rados.0.smithi067.stdout:32: finishing write tid 2 to smithi06731802-32 2024-03-30T12:55:47.579 INFO:tasks.rados.rados.0.smithi067.stdout:32: finishing write tid 3 to smithi06731802-32 2024-03-30T12:55:47.579 INFO:tasks.rados.rados.0.smithi067.stdout:32: finishing write tid 4 to smithi06731802-32 2024-03-30T12:55:47.579 INFO:tasks.rados.rados.0.smithi067.stdout:32: oid 32 updating version 0 to 6 2024-03-30T12:55:47.579 INFO:tasks.rados.rados.0.smithi067.stdout:32: oid 32 version 6 is already newer than 5 2024-03-30T12:55:47.579 INFO:tasks.rados.rados.0.smithi067.stdout:32: oid 32 version 6 is already newer than 4 2024-03-30T12:55:47.579 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 32 v 6 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-03-30T12:55:47.579 INFO:tasks.rados.rados.0.smithi067.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-03-30T12:55:47.579 INFO:tasks.rados.rados.0.smithi067.stdout:33: finishing write tid 1 to smithi06731802-33 2024-03-30T12:55:47.579 INFO:tasks.rados.rados.0.smithi067.stdout:32: done (15 left) 2024-03-30T12:55:47.579 INFO:tasks.rados.rados.0.smithi067.stdout:46: write initial oid 46 2024-03-30T12:55:47.579 INFO:tasks.rados.rados.0.smithi067.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2024-03-30T12:55:47.586 INFO:tasks.rados.rados.0.smithi067.stdout:46: writing smithi06731802-46 from 756315 to 1524285 tid 1 2024-03-30T12:55:47.594 INFO:tasks.rados.rados.0.smithi067.stdout:46: writing smithi06731802-46 from 1933572 to 2725242 tid 2 2024-03-30T12:55:47.594 INFO:tasks.rados.rados.0.smithi067.stdout: waiting on 16 2024-03-30T12:55:47.594 INFO:tasks.rados.rados.0.smithi067.stdout:33: finishing write tid 2 to smithi06731802-33 2024-03-30T12:55:47.594 INFO:tasks.rados.rados.0.smithi067.stdout:33: finishing write tid 3 to smithi06731802-33 2024-03-30T12:55:47.594 INFO:tasks.rados.rados.0.smithi067.stdout:33: finishing write tid 4 to smithi06731802-33 2024-03-30T12:55:47.594 INFO:tasks.rados.rados.0.smithi067.stdout:33: oid 33 updating version 0 to 3 2024-03-30T12:55:47.594 INFO:tasks.rados.rados.0.smithi067.stdout:33: oid 33 version 3 is already newer than 1 2024-03-30T12:55:47.595 INFO:tasks.rados.rados.0.smithi067.stdout:33: oid 33 version 3 is already newer than 2 2024-03-30T12:55:47.595 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 33 v 3 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-03-30T12:55:47.595 INFO:tasks.rados.rados.0.smithi067.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-03-30T12:55:47.595 INFO:tasks.rados.rados.0.smithi067.stdout:34: finishing write tid 1 to smithi06731802-34 2024-03-30T12:55:47.595 INFO:tasks.rados.rados.0.smithi067.stdout:33: done (15 left) 2024-03-30T12:55:47.595 INFO:tasks.rados.rados.0.smithi067.stdout:47: write initial oid 47 2024-03-30T12:55:47.595 INFO:tasks.rados.rados.0.smithi067.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2024-03-30T12:55:47.595 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-03-30T12:55:47.593+0000 7f710ac50640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25189) UID: 0 2024-03-30T12:55:47.600 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T12:55:47.601 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T12:55:47.601 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T12:55:47.601 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T12:55:47.601 INFO:tasks.rados.rados.0.smithi067.stdout:47: writing smithi06731802-47 from 773122 to 1332694 tid 1 2024-03-30T12:55:47.609 INFO:tasks.rados.rados.0.smithi067.stdout:47: writing smithi06731802-47 from 1992054 to 2643735 tid 2 2024-03-30T12:55:47.609 INFO:tasks.rados.rados.0.smithi067.stdout: waiting on 16 2024-03-30T12:55:47.609 INFO:tasks.rados.rados.0.smithi067.stdout:34: finishing write tid 2 to smithi06731802-34 2024-03-30T12:55:47.609 INFO:tasks.rados.rados.0.smithi067.stdout:34: finishing write tid 3 to smithi06731802-34 2024-03-30T12:55:47.610 INFO:tasks.rados.rados.0.smithi067.stdout:34: finishing write tid 4 to smithi06731802-34 2024-03-30T12:55:47.610 INFO:tasks.rados.rados.0.smithi067.stdout:34: oid 34 updating version 0 to 6 2024-03-30T12:55:47.610 INFO:tasks.rados.rados.0.smithi067.stdout:34: oid 34 version 6 is already newer than 4 2024-03-30T12:55:47.610 INFO:tasks.rados.rados.0.smithi067.stdout:34: oid 34 version 6 is already newer than 5 2024-03-30T12:55:47.610 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 34 v 6 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-03-30T12:55:47.610 INFO:tasks.rados.rados.0.smithi067.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-03-30T12:55:47.610 INFO:tasks.rados.rados.0.smithi067.stdout:31: finishing write tid 3 to smithi06731802-31 2024-03-30T12:55:47.610 INFO:tasks.rados.rados.0.smithi067.stdout:31: finishing write tid 4 to smithi06731802-31 2024-03-30T12:55:47.610 INFO:tasks.rados.rados.0.smithi067.stdout:31: oid 31 updating version 0 to 14 2024-03-30T12:55:47.610 INFO:tasks.rados.rados.0.smithi067.stdout:31: oid 31 updating version 14 to 15 2024-03-30T12:55:47.610 INFO:tasks.rados.rados.0.smithi067.stdout:31: oid 31 version 15 is already newer than 13 2024-03-30T12:55:47.610 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 31 v 15 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-03-30T12:55:47.610 INFO:tasks.rados.rados.0.smithi067.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-03-30T12:55:47.610 INFO:tasks.rados.rados.0.smithi067.stdout:35: finishing write tid 1 to smithi06731802-35 2024-03-30T12:55:47.611 INFO:tasks.rados.rados.0.smithi067.stdout:31: done (15 left) 2024-03-30T12:55:47.611 INFO:tasks.rados.rados.0.smithi067.stdout:34: done (14 left) 2024-03-30T12:55:47.611 INFO:tasks.rados.rados.0.smithi067.stdout:48: write initial oid 48 2024-03-30T12:55:47.611 INFO:tasks.rados.rados.0.smithi067.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2024-03-30T12:55:47.617 INFO:tasks.rados.rados.0.smithi067.stdout:48: writing smithi06731802-48 from 789929 to 1424750 tid 1 2024-03-30T12:55:47.619 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-03-30T12:55:47.625 INFO:tasks.rados.rados.0.smithi067.stdout:48: writing smithi06731802-48 from 2050536 to 2789929 tid 2 2024-03-30T12:55:47.625 INFO:tasks.rados.rados.0.smithi067.stdout:49: write initial oid 49 2024-03-30T12:55:47.625 INFO:tasks.rados.rados.0.smithi067.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2024-03-30T12:55:47.633 INFO:tasks.rados.rados.0.smithi067.stdout:49: writing smithi06731802-49 from 406736 to 1116806 tid 1 2024-03-30T12:55:47.639 INFO:tasks.rados.rados.0.smithi067.stdout:49: writing smithi06731802-49 from 1709018 to 2364368 tid 2 2024-03-30T12:55:47.639 INFO:tasks.rados.rados.0.smithi067.stdout: waiting on 16 2024-03-30T12:55:47.639 INFO:tasks.rados.rados.0.smithi067.stdout:35: finishing write tid 2 to smithi06731802-35 2024-03-30T12:55:47.639 INFO:tasks.rados.rados.0.smithi067.stdout:36: finishing write tid 1 to smithi06731802-36 2024-03-30T12:55:47.640 INFO:tasks.rados.rados.0.smithi067.stdout:36: finishing write tid 2 to smithi06731802-36 2024-03-30T12:55:47.640 INFO:tasks.rados.rados.0.smithi067.stdout:36: finishing write tid 3 to smithi06731802-36 2024-03-30T12:55:47.640 INFO:tasks.rados.rados.0.smithi067.stdout:36: finishing write tid 4 to smithi06731802-36 2024-03-30T12:55:47.640 INFO:tasks.rados.rados.0.smithi067.stdout:36: oid 36 updating version 0 to 11 2024-03-30T12:55:47.640 INFO:tasks.rados.rados.0.smithi067.stdout:36: oid 36 version 11 is already newer than 10 2024-03-30T12:55:47.640 INFO:tasks.rados.rados.0.smithi067.stdout:36: oid 36 updating version 11 to 12 2024-03-30T12:55:47.640 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 36 v 12 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-03-30T12:55:47.640 INFO:tasks.rados.rados.0.smithi067.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-03-30T12:55:47.640 INFO:tasks.rados.rados.0.smithi067.stdout:37: finishing write tid 1 to smithi06731802-37 2024-03-30T12:55:47.640 INFO:tasks.rados.rados.0.smithi067.stdout:37: finishing write tid 2 to smithi06731802-37 2024-03-30T12:55:47.640 INFO:tasks.rados.rados.0.smithi067.stdout:37: finishing write tid 3 to smithi06731802-37 2024-03-30T12:55:47.640 INFO:tasks.rados.rados.0.smithi067.stdout:37: finishing write tid 4 to smithi06731802-37 2024-03-30T12:55:47.640 INFO:tasks.rados.rados.0.smithi067.stdout:37: oid 37 updating version 0 to 8 2024-03-30T12:55:47.640 INFO:tasks.rados.rados.0.smithi067.stdout:37: oid 37 version 8 is already newer than 7 2024-03-30T12:55:47.640 INFO:tasks.rados.rados.0.smithi067.stdout:37: oid 37 updating version 8 to 9 2024-03-30T12:55:47.641 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 37 v 9 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-03-30T12:55:47.641 INFO:tasks.rados.rados.0.smithi067.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-03-30T12:55:47.641 INFO:tasks.rados.rados.0.smithi067.stdout:38: finishing write tid 1 to smithi06731802-38 2024-03-30T12:55:47.641 INFO:tasks.rados.rados.0.smithi067.stdout:36: done (15 left) 2024-03-30T12:55:47.641 INFO:tasks.rados.rados.0.smithi067.stdout:37: done (14 left) 2024-03-30T12:55:47.641 INFO:tasks.rados.rados.0.smithi067.stdout:50: write initial oid 50 2024-03-30T12:55:47.641 INFO:tasks.rados.rados.0.smithi067.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2024-03-30T12:55:47.646 INFO:tasks.rados.rados.0.smithi067.stdout:50: writing smithi06731802-50 from 423543 to 1208862 tid 1 2024-03-30T12:55:47.656 INFO:tasks.rados.rados.0.smithi067.stdout:50: writing smithi06731802-50 from 1767500 to 2566508 tid 2 2024-03-30T12:55:47.656 INFO:tasks.rados.rados.0.smithi067.stdout:51: read oid 1 snap -1 2024-03-30T12:55:47.656 INFO:tasks.rados.rados.0.smithi067.stdout:51: expect (ObjNum 0 snap 0 seq_num 0) 2024-03-30T12:55:47.656 INFO:tasks.rados.rados.0.smithi067.stdout: waiting on 16 2024-03-30T12:55:47.656 INFO:tasks.rados.rados.0.smithi067.stdout:35: finishing write tid 3 to smithi06731802-35 2024-03-30T12:55:47.656 INFO:tasks.rados.rados.0.smithi067.stdout:35: finishing write tid 4 to smithi06731802-35 2024-03-30T12:55:47.656 INFO:tasks.rados.rados.0.smithi067.stdout:35: oid 35 updating version 0 to 18 2024-03-30T12:55:47.656 INFO:tasks.rados.rados.0.smithi067.stdout:35: oid 35 version 18 is already newer than 16 2024-03-30T12:55:47.657 INFO:tasks.rados.rados.0.smithi067.stdout:35: oid 35 version 18 is already newer than 17 2024-03-30T12:55:47.657 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 35 v 18 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-03-30T12:55:47.657 INFO:tasks.rados.rados.0.smithi067.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-03-30T12:55:47.657 INFO:tasks.rados.rados.0.smithi067.stdout:38: finishing write tid 2 to smithi06731802-38 2024-03-30T12:55:47.657 INFO:tasks.rados.rados.0.smithi067.stdout:35: done (15 left) 2024-03-30T12:55:47.657 INFO:tasks.rados.rados.0.smithi067.stdout:52: snap_create 2024-03-30T12:55:47.657 INFO:tasks.rados.rados.0.smithi067.stdout:38: finishing write tid 3 to smithi06731802-38 2024-03-30T12:55:47.657 INFO:tasks.rados.rados.0.smithi067.stdout:38: finishing write tid 4 to smithi06731802-38 2024-03-30T12:55:47.657 INFO:tasks.rados.rados.0.smithi067.stdout:38: oid 38 updating version 0 to 5 2024-03-30T12:55:47.657 INFO:tasks.rados.rados.0.smithi067.stdout:38: oid 38 version 5 is already newer than 4 2024-03-30T12:55:47.657 INFO:tasks.rados.rados.0.smithi067.stdout:38: oid 38 updating version 5 to 6 2024-03-30T12:55:47.657 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 38 v 6 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-03-30T12:55:47.657 INFO:tasks.rados.rados.0.smithi067.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-03-30T12:55:47.657 INFO:tasks.rados.rados.0.smithi067.stdout:39: finishing write tid 1 to smithi06731802-39 2024-03-30T12:55:47.657 INFO:tasks.rados.rados.0.smithi067.stdout:39: finishing write tid 2 to smithi06731802-39 2024-03-30T12:55:47.658 INFO:tasks.rados.rados.0.smithi067.stdout:40: finishing write tid 1 to smithi06731802-40 2024-03-30T12:55:47.658 INFO:tasks.rados.rados.0.smithi067.stdout:40: finishing write tid 2 to smithi06731802-40 2024-03-30T12:55:47.658 INFO:tasks.rados.rados.0.smithi067.stdout:40: finishing write tid 3 to smithi06731802-40 2024-03-30T12:55:47.658 INFO:tasks.rados.rados.0.smithi067.stdout:40: finishing write tid 4 to smithi06731802-40 2024-03-30T12:55:47.658 INFO:tasks.rados.rados.0.smithi067.stdout:40: oid 40 updating version 0 to 1 2024-03-30T12:55:47.658 INFO:tasks.rados.rados.0.smithi067.stdout:40: oid 40 updating version 1 to 3 2024-03-30T12:55:47.658 INFO:tasks.rados.rados.0.smithi067.stdout:40: oid 40 version 3 is already newer than 2 2024-03-30T12:55:47.658 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 40 v 3 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-03-30T12:55:47.658 INFO:tasks.rados.rados.0.smithi067.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-03-30T12:55:47.658 INFO:tasks.rados.rados.0.smithi067.stdout:27: finishing write tid 4 to smithi06731802-27 2024-03-30T12:55:47.658 INFO:tasks.rados.rados.0.smithi067.stdout:27: oid 27 updating version 0 to 11 2024-03-30T12:55:47.658 INFO:tasks.rados.rados.0.smithi067.stdout:27: oid 27 updating version 11 to 12 2024-03-30T12:55:47.658 INFO:tasks.rados.rados.0.smithi067.stdout:27: oid 27 version 12 is already newer than 10 2024-03-30T12:55:47.658 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 27 v 12 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-03-30T12:55:47.659 INFO:tasks.rados.rados.0.smithi067.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-03-30T12:55:47.659 INFO:tasks.rados.rados.0.smithi067.stdout:41: finishing write tid 1 to smithi06731802-41 2024-03-30T12:55:47.659 INFO:tasks.rados.rados.0.smithi067.stdout:41: finishing write tid 2 to smithi06731802-41 2024-03-30T12:55:47.659 INFO:tasks.rados.rados.0.smithi067.stdout:41: finishing write tid 3 to smithi06731802-41 2024-03-30T12:55:47.659 INFO:tasks.rados.rados.0.smithi067.stdout:42: finishing write tid 1 to smithi06731802-42 2024-03-30T12:55:47.659 INFO:tasks.rados.rados.0.smithi067.stdout:43: finishing write tid 1 to smithi06731802-43 2024-03-30T12:55:47.659 INFO:tasks.rados.rados.0.smithi067.stdout:42: finishing write tid 2 to smithi06731802-42 2024-03-30T12:55:47.659 INFO:tasks.rados.rados.0.smithi067.stdout:42: finishing write tid 3 to smithi06731802-42 2024-03-30T12:55:47.659 INFO:tasks.rados.rados.0.smithi067.stdout:42: finishing write tid 4 to smithi06731802-42 2024-03-30T12:55:47.659 INFO:tasks.rados.rados.0.smithi067.stdout:42: oid 42 updating version 0 to 14 2024-03-30T12:55:47.659 INFO:tasks.rados.rados.0.smithi067.stdout:42: oid 42 version 14 is already newer than 13 2024-03-30T12:55:47.659 INFO:tasks.rados.rados.0.smithi067.stdout:42: oid 42 updating version 14 to 15 2024-03-30T12:55:47.659 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 42 v 15 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-03-30T12:55:47.659 INFO:tasks.rados.rados.0.smithi067.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-03-30T12:55:47.659 INFO:tasks.rados.rados.0.smithi067.stdout:43: finishing write tid 2 to smithi06731802-43 2024-03-30T12:55:47.660 INFO:tasks.rados.rados.0.smithi067.stdout:43: finishing write tid 3 to smithi06731802-43 2024-03-30T12:55:47.660 INFO:tasks.rados.rados.0.smithi067.stdout:43: finishing write tid 4 to smithi06731802-43 2024-03-30T12:55:47.660 INFO:tasks.rados.rados.0.smithi067.stdout:43: oid 43 updating version 0 to 9 2024-03-30T12:55:47.660 INFO:tasks.rados.rados.0.smithi067.stdout:43: oid 43 version 9 is already newer than 8 2024-03-30T12:55:47.660 INFO:tasks.rados.rados.0.smithi067.stdout:43: oid 43 version 9 is already newer than 7 2024-03-30T12:55:47.660 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 43 v 9 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-03-30T12:55:47.660 INFO:tasks.rados.rados.0.smithi067.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-03-30T12:55:47.660 INFO:tasks.rados.rados.0.smithi067.stdout:44: finishing write tid 1 to smithi06731802-44 2024-03-30T12:55:47.660 INFO:tasks.rados.rados.0.smithi067.stdout:44: finishing write tid 2 to smithi06731802-44 2024-03-30T12:55:47.660 INFO:tasks.rados.rados.0.smithi067.stdout:44: finishing write tid 3 to smithi06731802-44 2024-03-30T12:55:47.660 INFO:tasks.rados.rados.0.smithi067.stdout:44: finishing write tid 4 to smithi06731802-44 2024-03-30T12:55:47.660 INFO:tasks.rados.rados.0.smithi067.stdout:44: oid 44 updating version 0 to 9 2024-03-30T12:55:47.660 INFO:tasks.rados.rados.0.smithi067.stdout:44: oid 44 version 9 is already newer than 7 2024-03-30T12:55:47.660 INFO:tasks.rados.rados.0.smithi067.stdout:44: oid 44 version 9 is already newer than 8 2024-03-30T12:55:47.660 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 44 v 9 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-03-30T12:55:47.661 INFO:tasks.rados.rados.0.smithi067.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-03-30T12:55:47.661 INFO:tasks.rados.rados.0.smithi067.stdout:45: finishing write tid 1 to smithi06731802-45 2024-03-30T12:55:47.661 INFO:tasks.rados.rados.0.smithi067.stdout:45: finishing write tid 2 to smithi06731802-45 2024-03-30T12:55:47.661 INFO:tasks.rados.rados.0.smithi067.stdout:45: finishing write tid 3 to smithi06731802-45 2024-03-30T12:55:47.661 INFO:tasks.rados.rados.0.smithi067.stdout:45: finishing write tid 4 to smithi06731802-45 2024-03-30T12:55:47.661 INFO:tasks.rados.rados.0.smithi067.stdout:45: oid 45 updating version 0 to 18 2024-03-30T12:55:47.661 INFO:tasks.rados.rados.0.smithi067.stdout:45: oid 45 version 18 is already newer than 16 2024-03-30T12:55:47.661 INFO:tasks.rados.rados.0.smithi067.stdout:45: oid 45 version 18 is already newer than 17 2024-03-30T12:55:47.661 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 45 v 18 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-03-30T12:55:47.661 INFO:tasks.rados.rados.0.smithi067.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-03-30T12:55:47.661 INFO:tasks.rados.rados.0.smithi067.stdout:46: finishing write tid 1 to smithi06731802-46 2024-03-30T12:55:47.661 INFO:tasks.rados.rados.0.smithi067.stdout:46: finishing write tid 2 to smithi06731802-46 2024-03-30T12:55:47.661 INFO:tasks.rados.rados.0.smithi067.stdout:46: finishing write tid 3 to smithi06731802-46 2024-03-30T12:55:47.661 INFO:tasks.rados.rados.0.smithi067.stdout:46: finishing write tid 4 to smithi06731802-46 2024-03-30T12:55:47.661 INFO:tasks.rados.rados.0.smithi067.stdout:46: oid 46 updating version 0 to 5 2024-03-30T12:55:47.662 INFO:tasks.rados.rados.0.smithi067.stdout:46: oid 46 version 5 is already newer than 4 2024-03-30T12:55:47.662 INFO:tasks.rados.rados.0.smithi067.stdout:46: oid 46 updating version 5 to 6 2024-03-30T12:55:47.662 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 46 v 6 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-03-30T12:55:47.662 INFO:tasks.rados.rados.0.smithi067.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-03-30T12:55:47.662 INFO:tasks.rados.rados.0.smithi067.stdout:47: finishing write tid 1 to smithi06731802-47 2024-03-30T12:55:47.662 INFO:tasks.rados.rados.0.smithi067.stdout:47: finishing write tid 2 to smithi06731802-47 2024-03-30T12:55:47.662 INFO:tasks.rados.rados.0.smithi067.stdout:47: finishing write tid 3 to smithi06731802-47 2024-03-30T12:55:47.662 INFO:tasks.rados.rados.0.smithi067.stdout:47: finishing write tid 4 to smithi06731802-47 2024-03-30T12:55:47.662 INFO:tasks.rados.rados.0.smithi067.stdout:47: oid 47 updating version 0 to 11 2024-03-30T12:55:47.662 INFO:tasks.rados.rados.0.smithi067.stdout:47: oid 47 updating version 11 to 12 2024-03-30T12:55:47.662 INFO:tasks.rados.rados.0.smithi067.stdout:47: oid 47 version 12 is already newer than 10 2024-03-30T12:55:47.662 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 47 v 12 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-03-30T12:55:47.663 INFO:tasks.rados.rados.0.smithi067.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-03-30T12:55:47.663 INFO:tasks.rados.rados.0.smithi067.stdout:41: finishing write tid 4 to smithi06731802-41 2024-03-30T12:55:47.663 INFO:tasks.rados.rados.0.smithi067.stdout:41: oid 41 updating version 0 to 13 2024-03-30T12:55:47.663 INFO:tasks.rados.rados.0.smithi067.stdout:41: oid 41 updating version 13 to 15 2024-03-30T12:55:47.663 INFO:tasks.rados.rados.0.smithi067.stdout:41: oid 41 version 15 is already newer than 14 2024-03-30T12:55:47.663 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 41 v 15 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-03-30T12:55:47.663 INFO:tasks.rados.rados.0.smithi067.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-03-30T12:55:47.663 INFO:tasks.rados.rados.0.smithi067.stdout:48: finishing write tid 1 to smithi06731802-48 2024-03-30T12:55:47.663 INFO:tasks.rados.rados.0.smithi067.stdout:48: finishing write tid 2 to smithi06731802-48 2024-03-30T12:55:47.663 INFO:tasks.rados.rados.0.smithi067.stdout:39: finishing write tid 3 to smithi06731802-39 2024-03-30T12:55:47.663 INFO:tasks.rados.rados.0.smithi067.stdout:39: finishing write tid 4 to smithi06731802-39 2024-03-30T12:55:47.663 INFO:tasks.rados.rados.0.smithi067.stdout:39: oid 39 updating version 0 to 19 2024-03-30T12:55:47.663 INFO:tasks.rados.rados.0.smithi067.stdout:39: oid 39 updating version 19 to 21 2024-03-30T12:55:47.663 INFO:tasks.rados.rados.0.smithi067.stdout:39: oid 39 version 21 is already newer than 20 2024-03-30T12:55:47.664 INFO:tasks.rados.rados.0.smithi067.stdout:update_object_version oid 39 v 21 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-03-30T12:55:47.664 INFO:tasks.rados.rados.0.smithi067.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-03-30T12:55:47.664 INFO:tasks.rados.rados.0.smithi067.stdout:49: finishing write tid 1 to smithi06731802-49 2024-03-30T12:55:47.664 INFO:tasks.rados.rados.0.smithi067.stdout:49: finishing write tid 2 to smithi06731802-49 2024-03-30T12:55:47.664 INFO:tasks.rados.rados.0.smithi067.stdout:50: finishing write tid 1 to smithi06731802-50 2024-03-30T12:55:47.664 INFO:tasks.rados.rados.0.smithi067.stdout:50: finishing write tid 2 to smithi06731802-50 2024-03-30T12:55:47.668 INFO:tasks.rados.rados.0.smithi067.stdout:51: expect (ObjNum 0 snap 0 seq_num 0) 2024-03-30T12:55:47.695 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T12:55:47.691+0000 7f3b265f9640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25275) UID: 0 2024-03-30T12:55:47.742 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T12:55:47.742 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T12:55:47.742 INFO:teuthology.orchestra.run.smithi082.stdout: "complaint_time": 30, 2024-03-30T12:55:47.743 INFO:teuthology.orchestra.run.smithi082.stdout: "num_blocked_ops": 0 2024-03-30T12:55:47.743 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T12:55:47.754 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-30T12:55:47.754 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa/tasks/rados.py", line 280, in thread run.wait(tests.values()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' 2024-03-30T12:55:47.757 ERROR:teuthology.run_tasks:Manager failed: rados Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa/tasks/rados.py", line 299, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa/tasks/rados.py", line 280, in thread run.wait(tests.values()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' 2024-03-30T12:55:47.758 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-03-30T12:55:47.770 INFO:tasks.thrashosds:joining thrashosds 2024-03-30T12:55:47.771 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-03-30T12:55:47.940 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T12:55:47.940 INFO:teuthology.orchestra.run.smithi082.stdout: "size": 20, 2024-03-30T12:55:47.941 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 600, 2024-03-30T12:55:47.941 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [ 2024-03-30T12:55:47.941 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.941 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4433.0:166 1.2s0 1:4dd67a38:::smithi06731802-42:head [write 1699644~784332] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-03-30T12:55:47.941 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-03-30T12:55:47.536048+0000", 2024-03-30T12:55:47.941 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.40402495300000002, 2024-03-30T12:55:47.941 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.01489621, 2024-03-30T12:55:47.941 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-03-30T12:55:47.941 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T12:55:47.941 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-03-30T12:55:47.941 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4433", 2024-03-30T12:55:47.941 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.67:0/1327747904", 2024-03-30T12:55:47.941 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 166 2024-03-30T12:55:47.942 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.942 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-03-30T12:55:47.942 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.942 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-03-30T12:55:47.942 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.536048+0000", 2024-03-30T12:55:47.942 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:47.942 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.942 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.942 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-03-30T12:55:47.942 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.536048+0000", 2024-03-30T12:55:47.942 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:47.942 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.942 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.942 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-03-30T12:55:47.942 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.536049+0000", 2024-03-30T12:55:47.943 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.099e-06 2024-03-30T12:55:47.943 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.943 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.943 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-03-30T12:55:47.943 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.536938+0000", 2024-03-30T12:55:47.943 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000888853 2024-03-30T12:55:47.943 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.943 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.943 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-03-30T12:55:47.943 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.536952+0000", 2024-03-30T12:55:47.943 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4617000000000001e-05 2024-03-30T12:55:47.943 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.943 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.943 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-03-30T12:55:47.944 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.536962+0000", 2024-03-30T12:55:47.944 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.6579999999999997e-06 2024-03-30T12:55:47.944 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.944 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.944 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-03-30T12:55:47.944 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.537019+0000", 2024-03-30T12:55:47.944 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7080000000000002e-05 2024-03-30T12:55:47.944 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.944 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.944 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-03-30T12:55:47.944 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.537143+0000", 2024-03-30T12:55:47.944 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012403699999999999 2024-03-30T12:55:47.944 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.944 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.944 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_started", 2024-03-30T12:55:47.945 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.538209+0000", 2024-03-30T12:55:47.945 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001065631 2024-03-30T12:55:47.945 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.945 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.945 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_committed", 2024-03-30T12:55:47.945 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.540503+0000", 2024-03-30T12:55:47.945 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0022946609999999999 2024-03-30T12:55:47.945 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.945 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.945 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-03-30T12:55:47.945 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.550853+0000", 2024-03-30T12:55:47.945 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010349496999999999 2024-03-30T12:55:47.945 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.945 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.946 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-03-30T12:55:47.946 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.550944+0000", 2024-03-30T12:55:47.946 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.1076999999999996e-05 2024-03-30T12:55:47.946 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:47.946 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-03-30T12:55:47.946 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:47.946 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.946 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.946 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4433.0:167 1.2s0 1:4dd67a38:::smithi06731802-42:head [setxattr _header (56) in=7b,truncate 2689087] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-03-30T12:55:47.946 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-03-30T12:55:47.536980+0000", 2024-03-30T12:55:47.946 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.40309232099999998, 2024-03-30T12:55:47.946 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.014352205, 2024-03-30T12:55:47.946 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-03-30T12:55:47.946 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T12:55:47.947 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-03-30T12:55:47.947 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4433", 2024-03-30T12:55:47.947 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.67:0/1327747904", 2024-03-30T12:55:47.947 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 167 2024-03-30T12:55:47.947 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.947 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-03-30T12:55:47.947 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.947 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-03-30T12:55:47.947 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.536980+0000", 2024-03-30T12:55:47.947 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:47.947 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.947 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.947 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-03-30T12:55:47.947 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.536980+0000", 2024-03-30T12:55:47.947 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:47.948 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.948 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.948 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-03-30T12:55:47.948 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.536981+0000", 2024-03-30T12:55:47.948 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.64e-07 2024-03-30T12:55:47.948 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.948 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.948 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-03-30T12:55:47.948 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.536987+0000", 2024-03-30T12:55:47.948 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.164e-06 2024-03-30T12:55:47.948 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.948 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.948 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-03-30T12:55:47.948 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.536994+0000", 2024-03-30T12:55:47.948 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5329999999999997e-06 2024-03-30T12:55:47.949 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.949 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.949 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-03-30T12:55:47.949 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.537001+0000", 2024-03-30T12:55:47.949 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8889999999999996e-06 2024-03-30T12:55:47.949 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.949 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.949 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-03-30T12:55:47.949 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.538978+0000", 2024-03-30T12:55:47.949 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0019769169999999999 2024-03-30T12:55:47.949 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.949 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.949 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-03-30T12:55:47.949 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.539150+0000", 2024-03-30T12:55:47.949 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000172533 2024-03-30T12:55:47.950 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.950 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.950 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_started", 2024-03-30T12:55:47.950 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.540002+0000", 2024-03-30T12:55:47.950 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00085125100000000003 2024-03-30T12:55:47.950 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.950 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.950 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_committed", 2024-03-30T12:55:47.950 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.540736+0000", 2024-03-30T12:55:47.950 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00073442600000000005 2024-03-30T12:55:47.950 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.950 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.950 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-03-30T12:55:47.950 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.551239+0000", 2024-03-30T12:55:47.951 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010502401 2024-03-30T12:55:47.951 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.951 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.951 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-03-30T12:55:47.951 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.551333+0000", 2024-03-30T12:55:47.951 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.4127000000000002e-05 2024-03-30T12:55:47.951 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:47.951 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-03-30T12:55:47.951 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:47.951 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.951 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.951 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4433.0:168 1.2s0 1:4dd67a38:::smithi06731802-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2024-03-30T12:55:47.951 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-03-30T12:55:47.537009+0000", 2024-03-30T12:55:47.951 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.40306379599999997, 2024-03-30T12:55:47.951 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.015595039999999999, 2024-03-30T12:55:47.952 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-03-30T12:55:47.952 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-03-30T12:55:47.952 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-03-30T12:55:47.952 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4433", 2024-03-30T12:55:47.952 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.67:0/1327747904", 2024-03-30T12:55:47.952 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 168 2024-03-30T12:55:47.952 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.952 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-03-30T12:55:47.952 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.952 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-03-30T12:55:47.952 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.537009+0000", 2024-03-30T12:55:47.952 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:47.952 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.952 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.952 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-03-30T12:55:47.953 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.537009+0000", 2024-03-30T12:55:47.953 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:47.953 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.953 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.953 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-03-30T12:55:47.953 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.537009+0000", 2024-03-30T12:55:47.953 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.1100000000000001e-07 2024-03-30T12:55:47.953 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.953 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.953 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-03-30T12:55:47.953 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.537012+0000", 2024-03-30T12:55:47.953 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.17e-06 2024-03-30T12:55:47.953 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.953 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.953 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-03-30T12:55:47.954 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.537019+0000", 2024-03-30T12:55:47.954 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6660000000000008e-06 2024-03-30T12:55:47.954 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.954 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.954 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-03-30T12:55:47.954 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.537024+0000", 2024-03-30T12:55:47.954 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.8169999999999998e-06 2024-03-30T12:55:47.954 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.954 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.954 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-03-30T12:55:47.954 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.540561+0000", 2024-03-30T12:55:47.954 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0035368790000000002 2024-03-30T12:55:47.954 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.954 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.955 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for rw locks", 2024-03-30T12:55:47.955 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.540718+0000", 2024-03-30T12:55:47.955 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00015734899999999999 2024-03-30T12:55:47.955 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.955 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.955 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-03-30T12:55:47.955 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.551670+0000", 2024-03-30T12:55:47.955 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.010951651999999999 2024-03-30T12:55:47.955 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.955 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.955 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-03-30T12:55:47.955 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.551831+0000", 2024-03-30T12:55:47.955 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000161161 2024-03-30T12:55:47.955 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.955 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.956 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-03-30T12:55:47.956 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.552604+0000", 2024-03-30T12:55:47.956 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00077293499999999998 2024-03-30T12:55:47.956 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:47.956 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-03-30T12:55:47.956 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:47.956 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.956 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.956 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDECSubOpWriteReply(1.2s0 22/19 ECSubWriteReply(tid=54, last_complete=22'13, committed=1, applied=1))", 2024-03-30T12:55:47.956 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-03-30T12:55:47.537164+0000", 2024-03-30T12:55:47.956 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.40290921800000001, 2024-03-30T12:55:47.956 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0033363659999999999, 2024-03-30T12:55:47.956 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-03-30T12:55:47.956 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-03-30T12:55:47.956 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-03-30T12:55:47.957 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.957 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-03-30T12:55:47.957 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.537164+0000", 2024-03-30T12:55:47.957 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:47.957 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.957 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.957 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-03-30T12:55:47.957 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.537164+0000", 2024-03-30T12:55:47.957 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:47.957 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.957 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.957 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-03-30T12:55:47.957 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.537165+0000", 2024-03-30T12:55:47.957 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.5900000000000005e-07 2024-03-30T12:55:47.957 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.958 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.958 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-03-30T12:55:47.958 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.537175+0000", 2024-03-30T12:55:47.958 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.004e-05 2024-03-30T12:55:47.958 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.958 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.958 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-03-30T12:55:47.958 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.537208+0000", 2024-03-30T12:55:47.958 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.3600999999999999e-05 2024-03-30T12:55:47.958 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.958 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.958 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-03-30T12:55:47.958 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.537217+0000", 2024-03-30T12:55:47.958 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6419999999999996e-06 2024-03-30T12:55:47.959 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.959 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.959 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-03-30T12:55:47.959 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.540486+0000", 2024-03-30T12:55:47.959 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0032689910000000002 2024-03-30T12:55:47.959 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.959 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.959 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-03-30T12:55:47.959 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.540500+0000", 2024-03-30T12:55:47.959 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4133e-05 2024-03-30T12:55:47.959 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:47.959 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-03-30T12:55:47.959 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:47.959 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.960 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.960 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4433.0:173 1.ds0 1:bd9527f7:::smithi06731802-44:head [write 722701~617472,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-03-30T12:55:47.960 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-03-30T12:55:47.559281+0000", 2024-03-30T12:55:47.960 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.38079216799999999, 2024-03-30T12:55:47.960 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0037930440000000002, 2024-03-30T12:55:47.960 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-03-30T12:55:47.960 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T12:55:47.960 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-03-30T12:55:47.960 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4433", 2024-03-30T12:55:47.960 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.67:0/1327747904", 2024-03-30T12:55:47.960 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 173 2024-03-30T12:55:47.960 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.960 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-03-30T12:55:47.960 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.960 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-03-30T12:55:47.961 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.559281+0000", 2024-03-30T12:55:47.961 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:47.961 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.961 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.961 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-03-30T12:55:47.961 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.559281+0000", 2024-03-30T12:55:47.961 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:47.961 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.961 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.961 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-03-30T12:55:47.961 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.559282+0000", 2024-03-30T12:55:47.961 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.637e-06 2024-03-30T12:55:47.961 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.961 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.962 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-03-30T12:55:47.962 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.559821+0000", 2024-03-30T12:55:47.962 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00053833500000000003 2024-03-30T12:55:47.962 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.962 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.962 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-03-30T12:55:47.962 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.559836+0000", 2024-03-30T12:55:47.962 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5797000000000001e-05 2024-03-30T12:55:47.962 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.962 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.962 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-03-30T12:55:47.962 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.559842+0000", 2024-03-30T12:55:47.962 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.0619999999999997e-06 2024-03-30T12:55:47.962 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.962 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.963 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-03-30T12:55:47.963 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.559889+0000", 2024-03-30T12:55:47.963 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7043999999999999e-05 2024-03-30T12:55:47.963 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.963 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.963 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-03-30T12:55:47.963 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.559994+0000", 2024-03-30T12:55:47.963 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010443800000000001 2024-03-30T12:55:47.963 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.963 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.963 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_started", 2024-03-30T12:55:47.963 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.560839+0000", 2024-03-30T12:55:47.963 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00084550300000000005 2024-03-30T12:55:47.963 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.963 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.964 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_committed", 2024-03-30T12:55:47.964 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.562221+0000", 2024-03-30T12:55:47.964 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0013817600000000001 2024-03-30T12:55:47.964 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.964 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.964 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-03-30T12:55:47.964 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.563005+0000", 2024-03-30T12:55:47.964 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00078335399999999999 2024-03-30T12:55:47.964 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.964 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.964 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-03-30T12:55:47.964 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.563074+0000", 2024-03-30T12:55:47.964 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.9114000000000003e-05 2024-03-30T12:55:47.964 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:47.965 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-03-30T12:55:47.965 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:47.965 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.965 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.965 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4433.0:174 1.ds0 1:bd9527f7:::smithi06731802-44:head [write 1816608~788001] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-03-30T12:55:47.965 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-03-30T12:55:47.566471+0000", 2024-03-30T12:55:47.965 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.373601984, 2024-03-30T12:55:47.965 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0043077569999999997, 2024-03-30T12:55:47.965 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-03-30T12:55:47.965 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T12:55:47.965 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-03-30T12:55:47.965 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4433", 2024-03-30T12:55:47.965 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.67:0/1327747904", 2024-03-30T12:55:47.965 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 174 2024-03-30T12:55:47.965 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.966 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-03-30T12:55:47.966 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.966 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-03-30T12:55:47.966 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.566471+0000", 2024-03-30T12:55:47.966 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:47.966 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.966 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.966 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-03-30T12:55:47.966 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.566471+0000", 2024-03-30T12:55:47.966 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:47.966 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.966 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.966 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-03-30T12:55:47.966 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.566472+0000", 2024-03-30T12:55:47.967 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.02e-06 2024-03-30T12:55:47.967 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.967 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.967 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-03-30T12:55:47.967 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.567190+0000", 2024-03-30T12:55:47.967 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00071787499999999998 2024-03-30T12:55:47.967 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.967 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.967 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-03-30T12:55:47.967 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.567202+0000", 2024-03-30T12:55:47.967 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.1824000000000001e-05 2024-03-30T12:55:47.967 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.967 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.967 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-03-30T12:55:47.968 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.567217+0000", 2024-03-30T12:55:47.968 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5012e-05 2024-03-30T12:55:47.968 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.968 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.968 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-03-30T12:55:47.968 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.567264+0000", 2024-03-30T12:55:47.968 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.7481999999999997e-05 2024-03-30T12:55:47.968 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.968 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.968 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-03-30T12:55:47.968 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.567339+0000", 2024-03-30T12:55:47.968 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5136999999999996e-05 2024-03-30T12:55:47.968 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.968 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.968 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_started", 2024-03-30T12:55:47.969 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.568120+0000", 2024-03-30T12:55:47.969 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00078065000000000003 2024-03-30T12:55:47.969 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.969 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.969 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_committed", 2024-03-30T12:55:47.969 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.570645+0000", 2024-03-30T12:55:47.969 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002525242 2024-03-30T12:55:47.969 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.969 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.969 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-03-30T12:55:47.969 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.570686+0000", 2024-03-30T12:55:47.969 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.0945999999999997e-05 2024-03-30T12:55:47.969 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.969 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.969 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-03-30T12:55:47.970 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.570779+0000", 2024-03-30T12:55:47.970 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.2569000000000001e-05 2024-03-30T12:55:47.970 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:47.970 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-03-30T12:55:47.970 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:47.970 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.970 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.970 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4433.0:175 1.ds0 1:bd9527f7:::smithi06731802-44:head [setxattr _header (56) in=7b,truncate 2722701] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-03-30T12:55:47.970 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-03-30T12:55:47.567234+0000", 2024-03-30T12:55:47.970 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.37283870099999999, 2024-03-30T12:55:47.970 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0051760879999999997, 2024-03-30T12:55:47.970 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-03-30T12:55:47.970 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T12:55:47.970 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-03-30T12:55:47.970 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4433", 2024-03-30T12:55:47.971 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.67:0/1327747904", 2024-03-30T12:55:47.971 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 175 2024-03-30T12:55:47.971 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.971 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-03-30T12:55:47.971 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.971 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-03-30T12:55:47.971 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.567234+0000", 2024-03-30T12:55:47.971 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:47.971 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.971 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.971 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-03-30T12:55:47.971 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.567234+0000", 2024-03-30T12:55:47.971 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:47.971 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.971 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.972 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-03-30T12:55:47.972 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.567235+0000", 2024-03-30T12:55:47.972 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7400000000000002e-07 2024-03-30T12:55:47.972 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.972 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.972 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-03-30T12:55:47.972 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.567239+0000", 2024-03-30T12:55:47.972 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4800000000000003e-06 2024-03-30T12:55:47.972 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.972 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.972 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-03-30T12:55:47.972 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.567247+0000", 2024-03-30T12:55:47.972 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.0940000000000008e-06 2024-03-30T12:55:47.972 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.972 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.972 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-03-30T12:55:47.973 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.567254+0000", 2024-03-30T12:55:47.973 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.8499999999999996e-06 2024-03-30T12:55:47.973 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.973 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.973 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-03-30T12:55:47.973 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.569248+0000", 2024-03-30T12:55:47.973 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001994151 2024-03-30T12:55:47.973 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.973 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.973 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-03-30T12:55:47.973 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.569360+0000", 2024-03-30T12:55:47.973 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00011110100000000001 2024-03-30T12:55:47.973 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.973 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.973 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_started", 2024-03-30T12:55:47.974 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.569981+0000", 2024-03-30T12:55:47.974 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00062185099999999998 2024-03-30T12:55:47.974 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.974 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.974 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_committed", 2024-03-30T12:55:47.974 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.572089+0000", 2024-03-30T12:55:47.974 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002107959 2024-03-30T12:55:47.974 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.974 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.974 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-03-30T12:55:47.974 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.572326+0000", 2024-03-30T12:55:47.974 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00023660200000000001 2024-03-30T12:55:47.974 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.974 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.975 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-03-30T12:55:47.975 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.572410+0000", 2024-03-30T12:55:47.975 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.4226000000000003e-05 2024-03-30T12:55:47.975 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:47.975 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-03-30T12:55:47.975 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:47.975 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.975 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.975 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4433.0:176 1.ds0 1:bd9527f7:::smithi06731802-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2024-03-30T12:55:47.975 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-03-30T12:55:47.567264+0000", 2024-03-30T12:55:47.975 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.37280871900000001, 2024-03-30T12:55:47.975 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.006048103, 2024-03-30T12:55:47.975 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-03-30T12:55:47.975 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-03-30T12:55:47.976 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-03-30T12:55:47.976 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4433", 2024-03-30T12:55:47.976 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.67:0/1327747904", 2024-03-30T12:55:47.976 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 176 2024-03-30T12:55:47.976 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.976 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-03-30T12:55:47.976 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.976 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-03-30T12:55:47.976 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.567264+0000", 2024-03-30T12:55:47.976 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:47.976 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.976 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.976 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-03-30T12:55:47.976 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.567264+0000", 2024-03-30T12:55:47.977 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:47.977 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.977 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.977 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-03-30T12:55:47.977 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.567265+0000", 2024-03-30T12:55:47.977 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.4900000000000001e-07 2024-03-30T12:55:47.977 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.977 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.977 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-03-30T12:55:47.977 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.567268+0000", 2024-03-30T12:55:47.977 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9879999999999998e-06 2024-03-30T12:55:47.977 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.977 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.977 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-03-30T12:55:47.977 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.567277+0000", 2024-03-30T12:55:47.978 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.8319999999999995e-06 2024-03-30T12:55:47.978 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.978 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.978 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-03-30T12:55:47.978 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.567283+0000", 2024-03-30T12:55:47.978 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.366e-06 2024-03-30T12:55:47.978 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.978 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.978 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-03-30T12:55:47.978 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.570453+0000", 2024-03-30T12:55:47.978 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0031701920000000001 2024-03-30T12:55:47.978 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.978 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.978 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for rw locks", 2024-03-30T12:55:47.978 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.570516+0000", 2024-03-30T12:55:47.979 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.3387999999999997e-05 2024-03-30T12:55:47.979 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.979 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.979 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-03-30T12:55:47.979 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.572639+0000", 2024-03-30T12:55:47.979 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021228240000000002 2024-03-30T12:55:47.979 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.979 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.979 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-03-30T12:55:47.979 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.572688+0000", 2024-03-30T12:55:47.979 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9267000000000001e-05 2024-03-30T12:55:47.979 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.979 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.979 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-03-30T12:55:47.979 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.573312+0000", 2024-03-30T12:55:47.980 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00062379700000000005 2024-03-30T12:55:47.980 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:47.980 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-03-30T12:55:47.980 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:47.980 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.980 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.980 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDECSubOpWriteReply(1.ds0 22/19 ECSubWriteReply(tid=61, last_complete=22'8, committed=1, applied=1))", 2024-03-30T12:55:47.980 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-03-30T12:55:47.569768+0000", 2024-03-30T12:55:47.980 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.37030458199999999, 2024-03-30T12:55:47.980 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001014556, 2024-03-30T12:55:47.980 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-03-30T12:55:47.980 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-03-30T12:55:47.980 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-03-30T12:55:47.980 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.981 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-03-30T12:55:47.981 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.569768+0000", 2024-03-30T12:55:47.981 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:47.981 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.981 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.981 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-03-30T12:55:47.981 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.569768+0000", 2024-03-30T12:55:47.981 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:47.981 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.981 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.981 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-03-30T12:55:47.981 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.569769+0000", 2024-03-30T12:55:47.981 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.004e-06 2024-03-30T12:55:47.981 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.981 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.982 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-03-30T12:55:47.982 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.569774+0000", 2024-03-30T12:55:47.982 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.3080000000000001e-06 2024-03-30T12:55:47.982 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.982 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.982 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-03-30T12:55:47.982 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.569804+0000", 2024-03-30T12:55:47.982 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0602999999999998e-05 2024-03-30T12:55:47.982 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.982 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.982 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-03-30T12:55:47.982 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.569813+0000", 2024-03-30T12:55:47.982 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6410000000000008e-06 2024-03-30T12:55:47.982 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.983 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.983 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-03-30T12:55:47.983 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.570633+0000", 2024-03-30T12:55:47.983 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00082054100000000004 2024-03-30T12:55:47.983 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.983 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.983 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-03-30T12:55:47.983 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.570783+0000", 2024-03-30T12:55:47.983 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00014945900000000001 2024-03-30T12:55:47.983 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:47.983 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-03-30T12:55:47.983 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:47.983 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.983 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.984 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDECSubOpWriteReply(1.ds0 22/19 ECSubWriteReply(tid=61, last_complete=22'8, committed=1, applied=1))", 2024-03-30T12:55:47.984 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-03-30T12:55:47.569771+0000", 2024-03-30T12:55:47.984 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.370301981, 2024-03-30T12:55:47.984 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00084316800000000004, 2024-03-30T12:55:47.984 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-03-30T12:55:47.984 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-03-30T12:55:47.984 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-03-30T12:55:47.984 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.984 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-03-30T12:55:47.984 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.569771+0000", 2024-03-30T12:55:47.984 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:47.984 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.984 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.984 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-03-30T12:55:47.984 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.569771+0000", 2024-03-30T12:55:47.985 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:47.985 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.985 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.985 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-03-30T12:55:47.985 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.569772+0000", 2024-03-30T12:55:47.985 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 9.9600000000000008e-07 2024-03-30T12:55:47.985 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.985 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.985 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-03-30T12:55:47.985 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.569780+0000", 2024-03-30T12:55:47.985 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6209999999999993e-06 2024-03-30T12:55:47.985 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.985 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.985 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-03-30T12:55:47.985 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.569809+0000", 2024-03-30T12:55:47.986 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.8225000000000002e-05 2024-03-30T12:55:47.986 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.986 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.986 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-03-30T12:55:47.986 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.569816+0000", 2024-03-30T12:55:47.986 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7130000000000005e-06 2024-03-30T12:55:47.986 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.986 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.986 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-03-30T12:55:47.986 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.570584+0000", 2024-03-30T12:55:47.986 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00076752899999999998 2024-03-30T12:55:47.986 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.986 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.986 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-03-30T12:55:47.986 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.570614+0000", 2024-03-30T12:55:47.987 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.0083999999999999e-05 2024-03-30T12:55:47.987 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:47.987 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-03-30T12:55:47.987 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:47.987 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.987 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.987 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4433.0:177 1.2s0 1:48a48781:::smithi06731802-45:head [write 739508~692721,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-03-30T12:55:47.987 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-03-30T12:55:47.574867+0000", 2024-03-30T12:55:47.987 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.36520552000000001, 2024-03-30T12:55:47.987 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0038263529999999998, 2024-03-30T12:55:47.987 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-03-30T12:55:47.987 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T12:55:47.987 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-03-30T12:55:47.987 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4433", 2024-03-30T12:55:47.987 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.67:0/1327747904", 2024-03-30T12:55:47.988 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 177 2024-03-30T12:55:47.988 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.988 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-03-30T12:55:47.988 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.988 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-03-30T12:55:47.988 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.574867+0000", 2024-03-30T12:55:47.988 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:47.988 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.988 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.988 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-03-30T12:55:47.988 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.574867+0000", 2024-03-30T12:55:47.988 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:47.988 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.988 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.988 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-03-30T12:55:47.989 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.574869+0000", 2024-03-30T12:55:47.989 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2589999999999999e-06 2024-03-30T12:55:47.989 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.989 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.989 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-03-30T12:55:47.989 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.575460+0000", 2024-03-30T12:55:47.989 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00059177100000000005 2024-03-30T12:55:47.989 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.989 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.989 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-03-30T12:55:47.989 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.575472+0000", 2024-03-30T12:55:47.989 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.2123999999999999e-05 2024-03-30T12:55:47.989 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.989 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.989 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-03-30T12:55:47.989 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.575478+0000", 2024-03-30T12:55:47.990 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.6729999999999997e-06 2024-03-30T12:55:47.990 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.990 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.990 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-03-30T12:55:47.990 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.575530+0000", 2024-03-30T12:55:47.990 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1974000000000001e-05 2024-03-30T12:55:47.990 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.990 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.990 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-03-30T12:55:47.990 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.575633+0000", 2024-03-30T12:55:47.990 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000103214 2024-03-30T12:55:47.990 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.990 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.990 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_started", 2024-03-30T12:55:47.990 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.576603+0000", 2024-03-30T12:55:47.991 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00096986299999999995 2024-03-30T12:55:47.991 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.991 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.991 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_committed", 2024-03-30T12:55:47.991 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.578344+0000", 2024-03-30T12:55:47.991 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0017409890000000001 2024-03-30T12:55:47.991 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.991 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.991 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-03-30T12:55:47.991 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.578636+0000", 2024-03-30T12:55:47.991 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000291666 2024-03-30T12:55:47.991 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.991 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.991 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-03-30T12:55:47.992 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.578694+0000", 2024-03-30T12:55:47.992 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.7819999999999999e-05 2024-03-30T12:55:47.992 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:47.992 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-03-30T12:55:47.992 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:47.992 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.992 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.992 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4433.0:178 1.2s0 1:48a48781:::smithi06731802-45:head [write 1875090~648012] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-03-30T12:55:47.992 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-03-30T12:55:47.581064+0000", 2024-03-30T12:55:47.992 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.35900911200000002, 2024-03-30T12:55:47.992 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0062713259999999998, 2024-03-30T12:55:47.992 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-03-30T12:55:47.992 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T12:55:47.992 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-03-30T12:55:47.992 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4433", 2024-03-30T12:55:47.993 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.67:0/1327747904", 2024-03-30T12:55:47.993 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 178 2024-03-30T12:55:47.993 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.993 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-03-30T12:55:47.993 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.993 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-03-30T12:55:47.993 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.581064+0000", 2024-03-30T12:55:47.993 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:47.993 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.993 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.993 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-03-30T12:55:47.993 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.581064+0000", 2024-03-30T12:55:47.993 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:47.993 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.993 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.994 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-03-30T12:55:47.994 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.581065+0000", 2024-03-30T12:55:47.994 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4160000000000001e-06 2024-03-30T12:55:47.994 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.994 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.994 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-03-30T12:55:47.994 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.581675+0000", 2024-03-30T12:55:47.994 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00061031099999999999 2024-03-30T12:55:47.994 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.994 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.994 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-03-30T12:55:47.994 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.581693+0000", 2024-03-30T12:55:47.994 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.7725999999999999e-05 2024-03-30T12:55:47.994 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.995 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.995 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-03-30T12:55:47.995 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.581701+0000", 2024-03-30T12:55:47.995 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.2199999999999992e-06 2024-03-30T12:55:47.995 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.995 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.995 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-03-30T12:55:47.995 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.581763+0000", 2024-03-30T12:55:47.995 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.1427999999999995e-05 2024-03-30T12:55:47.995 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.995 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.995 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-03-30T12:55:47.995 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.581886+0000", 2024-03-30T12:55:47.995 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00012337000000000001 2024-03-30T12:55:47.996 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.996 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.996 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_started", 2024-03-30T12:55:47.996 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.582892+0000", 2024-03-30T12:55:47.996 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0010063069999999999 2024-03-30T12:55:47.996 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.996 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.996 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_committed", 2024-03-30T12:55:47.996 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.586524+0000", 2024-03-30T12:55:47.996 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003631062 2024-03-30T12:55:47.996 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.996 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.996 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-03-30T12:55:47.996 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.587277+0000", 2024-03-30T12:55:47.997 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00075307399999999995 2024-03-30T12:55:47.997 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.997 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.997 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-03-30T12:55:47.997 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.587335+0000", 2024-03-30T12:55:47.997 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.8412000000000002e-05 2024-03-30T12:55:47.997 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:47.997 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-03-30T12:55:47.997 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:47.997 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.997 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.997 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4433.0:179 1.2s0 1:48a48781:::smithi06731802-45:head [setxattr _header (56) in=7b,truncate 2739508] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-03-30T12:55:47.997 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-03-30T12:55:47.581719+0000", 2024-03-30T12:55:47.997 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.35835358, 2024-03-30T12:55:47.997 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.005859398, 2024-03-30T12:55:47.998 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-03-30T12:55:47.998 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-30T12:55:47.998 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-03-30T12:55:47.998 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4433", 2024-03-30T12:55:47.998 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.67:0/1327747904", 2024-03-30T12:55:47.998 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 179 2024-03-30T12:55:47.998 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.998 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-03-30T12:55:47.998 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.998 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-03-30T12:55:47.998 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.581719+0000", 2024-03-30T12:55:47.998 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:47.998 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.998 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.999 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-03-30T12:55:47.999 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.581719+0000", 2024-03-30T12:55:47.999 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:47.999 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.999 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.999 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-03-30T12:55:47.999 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.581720+0000", 2024-03-30T12:55:47.999 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.8800000000000002e-07 2024-03-30T12:55:47.999 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.999 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.999 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-03-30T12:55:47.999 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.581724+0000", 2024-03-30T12:55:47.999 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.9899999999999999e-06 2024-03-30T12:55:47.999 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:47.999 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:47.999 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-03-30T12:55:48.000 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.581731+0000", 2024-03-30T12:55:48.000 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.7700000000000004e-06 2024-03-30T12:55:48.000 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.000 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.000 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-03-30T12:55:48.000 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.581736+0000", 2024-03-30T12:55:48.000 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.0839999999999999e-06 2024-03-30T12:55:48.000 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.000 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.000 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-03-30T12:55:48.000 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.583891+0000", 2024-03-30T12:55:48.000 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0021551909999999999 2024-03-30T12:55:48.000 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.000 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.000 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-03-30T12:55:48.001 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.583998+0000", 2024-03-30T12:55:48.001 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000106821 2024-03-30T12:55:48.001 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.001 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.001 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_started", 2024-03-30T12:55:48.001 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.584437+0000", 2024-03-30T12:55:48.001 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00043943299999999998 2024-03-30T12:55:48.001 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.001 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.001 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_committed", 2024-03-30T12:55:48.001 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.586929+0000", 2024-03-30T12:55:48.001 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0024915240000000002 2024-03-30T12:55:48.001 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.001 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.001 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "commit_sent", 2024-03-30T12:55:48.002 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.587474+0000", 2024-03-30T12:55:48.002 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00054569799999999995 2024-03-30T12:55:48.002 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.002 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.002 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-03-30T12:55:48.002 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.587579+0000", 2024-03-30T12:55:48.002 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00010409899999999999 2024-03-30T12:55:48.002 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:48.002 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-03-30T12:55:48.002 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:48.002 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.002 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.002 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "osd_op(client.4433.0:180 1.2s0 1:48a48781:::smithi06731802-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2024-03-30T12:55:48.002 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-03-30T12:55:47.581758+0000", 2024-03-30T12:55:48.002 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.35831516400000002, 2024-03-30T12:55:48.003 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0066524150000000001, 2024-03-30T12:55:48.003 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-03-30T12:55:48.003 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "started", 2024-03-30T12:55:48.003 INFO:teuthology.orchestra.run.smithi082.stdout: "client_info": { 2024-03-30T12:55:48.003 INFO:teuthology.orchestra.run.smithi082.stdout: "client": "client.4433", 2024-03-30T12:55:48.003 INFO:teuthology.orchestra.run.smithi082.stdout: "client_addr": "172.21.15.67:0/1327747904", 2024-03-30T12:55:48.003 INFO:teuthology.orchestra.run.smithi082.stdout: "tid": 180 2024-03-30T12:55:48.003 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.003 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-03-30T12:55:48.003 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.003 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-03-30T12:55:48.003 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.581758+0000", 2024-03-30T12:55:48.003 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:48.003 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.003 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.004 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-03-30T12:55:48.004 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.581758+0000", 2024-03-30T12:55:48.004 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:48.004 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.004 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.004 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-03-30T12:55:48.004 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.581758+0000", 2024-03-30T12:55:48.004 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6199999999999997e-07 2024-03-30T12:55:48.004 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.004 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.004 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-03-30T12:55:48.004 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.581761+0000", 2024-03-30T12:55:48.004 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.7240000000000001e-06 2024-03-30T12:55:48.004 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.004 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.005 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-03-30T12:55:48.005 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.581769+0000", 2024-03-30T12:55:48.005 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.7999999999999999e-06 2024-03-30T12:55:48.005 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.005 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.005 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-03-30T12:55:48.005 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.581774+0000", 2024-03-30T12:55:48.005 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.1460000000000004e-06 2024-03-30T12:55:48.005 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.005 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.005 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-03-30T12:55:48.005 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.584718+0000", 2024-03-30T12:55:48.005 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0029436699999999998 2024-03-30T12:55:48.005 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.005 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.005 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "waiting for rw locks", 2024-03-30T12:55:48.006 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.584898+0000", 2024-03-30T12:55:48.006 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018001200000000001 2024-03-30T12:55:48.006 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.006 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.006 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-03-30T12:55:48.006 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.587801+0000", 2024-03-30T12:55:48.006 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.002903573 2024-03-30T12:55:48.006 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.006 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.006 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "started", 2024-03-30T12:55:48.006 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.587851+0000", 2024-03-30T12:55:48.006 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 4.9811999999999997e-05 2024-03-30T12:55:48.006 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.006 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.007 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-03-30T12:55:48.007 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.588410+0000", 2024-03-30T12:55:48.007 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00055901599999999996 2024-03-30T12:55:48.007 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:48.007 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-03-30T12:55:48.007 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:48.007 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.007 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.007 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDECSubOpWriteReply(1.2s0 22/19 ECSubWriteReply(tid=68, last_complete=22'18, committed=1, applied=1))", 2024-03-30T12:55:48.007 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-03-30T12:55:47.587109+0000", 2024-03-30T12:55:48.007 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.35296411, 2024-03-30T12:55:48.007 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00066444699999999998, 2024-03-30T12:55:48.007 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-03-30T12:55:48.007 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-03-30T12:55:48.007 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-03-30T12:55:48.008 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.008 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-03-30T12:55:48.008 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.587109+0000", 2024-03-30T12:55:48.008 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:48.008 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.008 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.008 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-03-30T12:55:48.008 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.587109+0000", 2024-03-30T12:55:48.008 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:48.008 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.008 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.008 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-03-30T12:55:48.008 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.587109+0000", 2024-03-30T12:55:48.008 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6199999999999999e-07 2024-03-30T12:55:48.008 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.009 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.009 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-03-30T12:55:48.009 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.587111+0000", 2024-03-30T12:55:48.009 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.12e-06 2024-03-30T12:55:48.009 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.009 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.009 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-03-30T12:55:48.009 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.587188+0000", 2024-03-30T12:55:48.009 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.6822000000000004e-05 2024-03-30T12:55:48.009 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.009 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.009 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-03-30T12:55:48.009 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.587194+0000", 2024-03-30T12:55:48.009 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2779999999999999e-06 2024-03-30T12:55:48.009 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.009 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.010 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-03-30T12:55:48.010 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.587377+0000", 2024-03-30T12:55:48.010 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00018265800000000001 2024-03-30T12:55:48.010 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.010 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.010 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-03-30T12:55:48.010 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.587773+0000", 2024-03-30T12:55:48.010 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00039630699999999999 2024-03-30T12:55:48.010 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:48.010 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-03-30T12:55:48.010 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:48.010 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.010 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.010 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDECSubOpWrite(1.4s2 22/19 ECSubWrite(tid=32, reqid=client.4433.0:189, at_version=22'4, trim_to=0'0, roll_forward_to=22'3))", 2024-03-30T12:55:48.011 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-03-30T12:55:47.621938+0000", 2024-03-30T12:55:48.011 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.31813455899999998, 2024-03-30T12:55:48.011 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003330679, 2024-03-30T12:55:48.011 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-03-30T12:55:48.011 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-03-30T12:55:48.011 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-03-30T12:55:48.011 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.011 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-03-30T12:55:48.011 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.621938+0000", 2024-03-30T12:55:48.011 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:48.011 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.011 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.011 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-03-30T12:55:48.011 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.621938+0000", 2024-03-30T12:55:48.011 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:48.012 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.012 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.012 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-03-30T12:55:48.012 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.621941+0000", 2024-03-30T12:55:48.012 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5100000000000001e-06 2024-03-30T12:55:48.012 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.012 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.012 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-03-30T12:55:48.012 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.623448+0000", 2024-03-30T12:55:48.012 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015069180000000001 2024-03-30T12:55:48.012 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.012 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.012 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-03-30T12:55:48.012 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.623522+0000", 2024-03-30T12:55:48.012 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4668999999999999e-05 2024-03-30T12:55:48.012 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.013 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.013 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-03-30T12:55:48.013 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.623542+0000", 2024-03-30T12:55:48.013 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.9704e-05 2024-03-30T12:55:48.013 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.013 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.013 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-03-30T12:55:48.013 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.623630+0000", 2024-03-30T12:55:48.013 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.7860000000000002e-05 2024-03-30T12:55:48.013 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.013 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.013 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_started", 2024-03-30T12:55:48.013 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.623637+0000", 2024-03-30T12:55:48.013 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.4019999999999996e-06 2024-03-30T12:55:48.013 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.014 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.014 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_committed", 2024-03-30T12:55:48.014 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.625230+0000", 2024-03-30T12:55:48.014 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0015930300000000001 2024-03-30T12:55:48.014 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.014 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.014 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-03-30T12:55:48.014 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.625269+0000", 2024-03-30T12:55:48.014 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.8586e-05 2024-03-30T12:55:48.014 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:48.014 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-03-30T12:55:48.014 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:48.014 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.015 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.015 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDECSubOpWrite(1.4s2 22/19 ECSubWrite(tid=33, reqid=client.4433.0:189, at_version=0'0, trim_to=0'0, roll_forward_to=22'4))", 2024-03-30T12:55:48.015 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-03-30T12:55:47.625756+0000", 2024-03-30T12:55:48.015 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.31431705599999998, 2024-03-30T12:55:48.015 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00091680099999999997, 2024-03-30T12:55:48.015 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-03-30T12:55:48.015 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-03-30T12:55:48.015 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-03-30T12:55:48.015 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.015 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-03-30T12:55:48.015 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.625756+0000", 2024-03-30T12:55:48.015 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:48.015 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.016 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.016 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-03-30T12:55:48.016 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.625756+0000", 2024-03-30T12:55:48.016 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:48.016 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.016 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.016 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-03-30T12:55:48.016 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.625757+0000", 2024-03-30T12:55:48.016 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4959999999999999e-06 2024-03-30T12:55:48.016 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.016 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.016 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-03-30T12:55:48.016 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.625783+0000", 2024-03-30T12:55:48.016 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.5469999999999998e-05 2024-03-30T12:55:48.016 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.017 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.017 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-03-30T12:55:48.017 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.625859+0000", 2024-03-30T12:55:48.017 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.5828999999999995e-05 2024-03-30T12:55:48.017 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.017 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.017 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-03-30T12:55:48.017 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.625881+0000", 2024-03-30T12:55:48.017 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.2096999999999999e-05 2024-03-30T12:55:48.017 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.017 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.017 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-03-30T12:55:48.017 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.625947+0000", 2024-03-30T12:55:48.017 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.6793999999999998e-05 2024-03-30T12:55:48.018 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.018 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.018 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_started", 2024-03-30T12:55:48.018 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.625954+0000", 2024-03-30T12:55:48.018 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.2439999999999998e-06 2024-03-30T12:55:48.018 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.018 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.018 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_committed", 2024-03-30T12:55:48.018 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.626643+0000", 2024-03-30T12:55:48.018 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00068963900000000003 2024-03-30T12:55:48.018 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.018 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.018 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-03-30T12:55:48.018 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.626673+0000", 2024-03-30T12:55:48.018 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.9232000000000001e-05 2024-03-30T12:55:48.019 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:48.019 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-03-30T12:55:48.019 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:48.019 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.019 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.019 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDECSubOpWrite(1.4s2 22/19 ECSubWrite(tid=34, reqid=client.4433.0:190, at_version=22'5, trim_to=0'0, roll_forward_to=22'4))", 2024-03-30T12:55:48.019 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-03-30T12:55:47.629598+0000", 2024-03-30T12:55:48.019 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.31047467499999998, 2024-03-30T12:55:48.019 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.003371307, 2024-03-30T12:55:48.019 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-03-30T12:55:48.019 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-03-30T12:55:48.019 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-03-30T12:55:48.019 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.019 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-03-30T12:55:48.020 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.629598+0000", 2024-03-30T12:55:48.020 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:48.020 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.020 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.020 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-03-30T12:55:48.020 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.629598+0000", 2024-03-30T12:55:48.020 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:48.020 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.020 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.020 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-03-30T12:55:48.020 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.629600+0000", 2024-03-30T12:55:48.021 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.477e-06 2024-03-30T12:55:48.021 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.021 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.021 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-03-30T12:55:48.021 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.630836+0000", 2024-03-30T12:55:48.021 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001236233 2024-03-30T12:55:48.021 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.021 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.021 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-03-30T12:55:48.021 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.630947+0000", 2024-03-30T12:55:48.021 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000110913 2024-03-30T12:55:48.021 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.021 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.021 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-03-30T12:55:48.022 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.630971+0000", 2024-03-30T12:55:48.022 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.3986e-05 2024-03-30T12:55:48.022 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.022 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.022 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-03-30T12:55:48.022 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.631115+0000", 2024-03-30T12:55:48.022 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.000143876 2024-03-30T12:55:48.022 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.022 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.022 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_started", 2024-03-30T12:55:48.022 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.631136+0000", 2024-03-30T12:55:48.022 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.1009000000000002e-05 2024-03-30T12:55:48.022 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.022 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.022 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_committed", 2024-03-30T12:55:48.023 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.632933+0000", 2024-03-30T12:55:48.023 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0017974690000000001 2024-03-30T12:55:48.023 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.023 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.023 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-03-30T12:55:48.023 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.632969+0000", 2024-03-30T12:55:48.023 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.6344000000000003e-05 2024-03-30T12:55:48.023 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:48.023 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-03-30T12:55:48.023 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:48.023 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.023 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.023 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDECSubOpWrite(1.4s2 22/19 ECSubWrite(tid=35, reqid=client.4433.0:191, at_version=22'6, trim_to=0'0, roll_forward_to=22'4))", 2024-03-30T12:55:48.023 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-03-30T12:55:47.630843+0000", 2024-03-30T12:55:48.023 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.30922992500000002, 2024-03-30T12:55:48.024 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0024106900000000001, 2024-03-30T12:55:48.024 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-03-30T12:55:48.024 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-03-30T12:55:48.024 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-03-30T12:55:48.024 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.024 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-03-30T12:55:48.024 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.630843+0000", 2024-03-30T12:55:48.024 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:48.024 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.024 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.024 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-03-30T12:55:48.024 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.630843+0000", 2024-03-30T12:55:48.024 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:48.024 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.024 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.025 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-03-30T12:55:48.025 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.630844+0000", 2024-03-30T12:55:48.025 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 7.3300000000000001e-07 2024-03-30T12:55:48.025 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.025 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.025 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-03-30T12:55:48.025 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.630868+0000", 2024-03-30T12:55:48.025 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.4584e-05 2024-03-30T12:55:48.025 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.025 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.025 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-03-30T12:55:48.025 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.631031+0000", 2024-03-30T12:55:48.025 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00016291499999999999 2024-03-30T12:55:48.025 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.025 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.026 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-03-30T12:55:48.026 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.631047+0000", 2024-03-30T12:55:48.026 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.5603e-05 2024-03-30T12:55:48.026 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.026 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.026 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-03-30T12:55:48.026 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.631972+0000", 2024-03-30T12:55:48.026 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.00092500700000000004 2024-03-30T12:55:48.026 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.026 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.026 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_started", 2024-03-30T12:55:48.026 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.631980+0000", 2024-03-30T12:55:48.026 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.3550000000000005e-06 2024-03-30T12:55:48.026 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.026 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.027 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_committed", 2024-03-30T12:55:48.027 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.633227+0000", 2024-03-30T12:55:48.027 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.001247092 2024-03-30T12:55:48.027 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.027 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.027 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-03-30T12:55:48.027 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.633254+0000", 2024-03-30T12:55:48.027 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.6401e-05 2024-03-30T12:55:48.027 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:48.027 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-03-30T12:55:48.027 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:48.027 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.027 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.027 INFO:teuthology.orchestra.run.smithi082.stdout: "description": "MOSDECSubOpWrite(1.4s2 22/19 ECSubWrite(tid=36, reqid=client.4433.0:191, at_version=0'0, trim_to=0'0, roll_forward_to=22'6))", 2024-03-30T12:55:48.027 INFO:teuthology.orchestra.run.smithi082.stdout: "initiated_at": "2024-03-30T12:55:47.633718+0000", 2024-03-30T12:55:48.028 INFO:teuthology.orchestra.run.smithi082.stdout: "age": 0.30635519500000002, 2024-03-30T12:55:48.028 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0016981660000000001, 2024-03-30T12:55:48.028 INFO:teuthology.orchestra.run.smithi082.stdout: "type_data": { 2024-03-30T12:55:48.028 INFO:teuthology.orchestra.run.smithi082.stdout: "flag_point": "reached pg", 2024-03-30T12:55:48.028 INFO:teuthology.orchestra.run.smithi082.stdout: "events": [ 2024-03-30T12:55:48.028 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.028 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "initiated", 2024-03-30T12:55:48.028 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.633718+0000", 2024-03-30T12:55:48.028 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:48.028 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.028 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.028 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "header_read", 2024-03-30T12:55:48.028 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.633718+0000", 2024-03-30T12:55:48.028 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0 2024-03-30T12:55:48.028 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.029 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.029 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "throttled", 2024-03-30T12:55:48.029 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.633719+0000", 2024-03-30T12:55:48.029 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4139999999999999e-06 2024-03-30T12:55:48.029 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.029 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.029 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "all_read", 2024-03-30T12:55:48.029 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.633739+0000", 2024-03-30T12:55:48.029 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 2.0196000000000001e-05 2024-03-30T12:55:48.029 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.029 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.029 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "dispatched", 2024-03-30T12:55:48.029 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.633798+0000", 2024-03-30T12:55:48.029 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 5.9103e-05 2024-03-30T12:55:48.029 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.029 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.030 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "queued_for_pg", 2024-03-30T12:55:48.030 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.633813+0000", 2024-03-30T12:55:48.030 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 1.4667e-05 2024-03-30T12:55:48.030 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.030 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.030 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "reached_pg", 2024-03-30T12:55:48.030 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.633877+0000", 2024-03-30T12:55:48.030 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 6.4206000000000002e-05 2024-03-30T12:55:48.030 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.030 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.030 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_started", 2024-03-30T12:55:48.030 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.633886+0000", 2024-03-30T12:55:48.030 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 8.6819999999999992e-06 2024-03-30T12:55:48.030 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.030 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.031 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "sub_op_committed", 2024-03-30T12:55:48.031 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.635384+0000", 2024-03-30T12:55:48.031 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 0.0014981300000000001 2024-03-30T12:55:48.031 INFO:teuthology.orchestra.run.smithi082.stdout: }, 2024-03-30T12:55:48.031 INFO:teuthology.orchestra.run.smithi082.stdout: { 2024-03-30T12:55:48.031 INFO:teuthology.orchestra.run.smithi082.stdout: "event": "done", 2024-03-30T12:55:48.031 INFO:teuthology.orchestra.run.smithi082.stdout: "time": "2024-03-30T12:55:47.635416+0000", 2024-03-30T12:55:48.031 INFO:teuthology.orchestra.run.smithi082.stdout: "duration": 3.1767999999999998e-05 2024-03-30T12:55:48.031 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:48.031 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-03-30T12:55:48.031 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:48.031 INFO:teuthology.orchestra.run.smithi082.stdout: } 2024-03-30T12:55:48.031 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-03-30T12:55:48.031 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T12:55:48.964 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-03-30T12:55:49.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:49.089+0000 7fb5978c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:49.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:49.105+0000 7fb5978c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:49.969 INFO:teuthology.orchestra.run.smithi067.stderr:noscrub is unset 2024-03-30T12:55:49.994 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-03-30T12:55:50.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:50.117+0000 7f221b29b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:50.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:55:50.133+0000 7f221b29b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:55:50.966 INFO:teuthology.orchestra.run.smithi067.stderr:nodeep-scrub is unset 2024-03-30T12:56:40.123 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T12:56:40.123 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:56:40.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:56:40.247+0000 7f5e14b00640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:56:40.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:56:40.263+0000 7f5e14b00640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:56:40.438 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:56:40.438 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T12:56:40.456 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-03-30T12:56:39.800209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":103,"num_read_kb":4624,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":310212,"kb_used_data":94044,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754664508,"statfs":{"total":773094113280,"available":772776456192,"internally_reserved":0,"allocated":96301056,"data_stored":94969928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004259"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.193120+0000","last_change":"2024-03-30T12:55:43.931163+0000","last_active":"2024-03-30T12:55:52.193120+0000","last_peered":"2024-03-30T12:55:52.193120+0000","last_clean":"2024-03-30T12:55:52.193120+0000","last_became_active":"2024-03-30T12:55:43.930823+0000","last_became_peered":"2024-03-30T12:55:43.930823+0000","last_unstale":"2024-03-30T12:55:52.193120+0000","last_undegraded":"2024-03-30T12:55:52.193120+0000","last_fullsized":"2024-03-30T12:55:52.193120+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:54.174180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:50.973407+0000","last_change":"2024-03-30T12:55:43.930916+0000","last_active":"2024-03-30T12:55:50.973407+0000","last_peered":"2024-03-30T12:55:50.973407+0000","last_clean":"2024-03-30T12:55:50.973407+0000","last_became_active":"2024-03-30T12:55:43.930591+0000","last_became_peered":"2024-03-30T12:55:43.930591+0000","last_unstale":"2024-03-30T12:55:50.973407+0000","last_undegraded":"2024-03-30T12:55:50.973407+0000","last_fullsized":"2024-03-30T12:55:50.973407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:07.237338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":34,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.600194+0000","last_change":"2024-03-30T12:55:43.931416+0000","last_active":"2024-03-30T12:55:51.600194+0000","last_peered":"2024-03-30T12:55:51.600194+0000","last_clean":"2024-03-30T12:55:51.600194+0000","last_became_active":"2024-03-30T12:55:43.931053+0000","last_became_peered":"2024-03-30T12:55:43.931053+0000","last_unstale":"2024-03-30T12:55:51.600194+0000","last_undegraded":"2024-03-30T12:55:51.600194+0000","last_fullsized":"2024-03-30T12:55:51.600194+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:02.687981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.600376+0000","last_change":"2024-03-30T12:55:43.938795+0000","last_active":"2024-03-30T12:55:51.600376+0000","last_peered":"2024-03-30T12:55:51.600376+0000","last_clean":"2024-03-30T12:55:51.600376+0000","last_became_active":"2024-03-30T12:55:43.938430+0000","last_became_peered":"2024-03-30T12:55:43.938430+0000","last_unstale":"2024-03-30T12:55:51.600376+0000","last_undegraded":"2024-03-30T12:55:51.600376+0000","last_fullsized":"2024-03-30T12:55:51.600376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:44.051242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.951550+0000","last_change":"2024-03-30T12:55:43.934224+0000","last_active":"2024-03-30T12:55:51.951550+0000","last_peered":"2024-03-30T12:55:51.951550+0000","last_clean":"2024-03-30T12:55:51.951550+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T12:55:51.951550+0000","last_undegraded":"2024-03-30T12:55:51.951550+0000","last_fullsized":"2024-03-30T12:55:51.951550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:03.609885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":42,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.192795+0000","last_change":"2024-03-30T12:55:43.927603+0000","last_active":"2024-03-30T12:55:52.192795+0000","last_peered":"2024-03-30T12:55:52.192795+0000","last_clean":"2024-03-30T12:55:52.192795+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T12:55:52.192795+0000","last_undegraded":"2024-03-30T12:55:52.192795+0000","last_fullsized":"2024-03-30T12:55:52.192795+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:47.393835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":46,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.600181+0000","last_change":"2024-03-30T12:55:43.934166+0000","last_active":"2024-03-30T12:55:51.600181+0000","last_peered":"2024-03-30T12:55:51.600181+0000","last_clean":"2024-03-30T12:55:51.600181+0000","last_became_active":"2024-03-30T12:55:43.933833+0000","last_became_peered":"2024-03-30T12:55:43.933833+0000","last_unstale":"2024-03-30T12:55:51.600181+0000","last_undegraded":"2024-03-30T12:55:51.600181+0000","last_fullsized":"2024-03-30T12:55:51.600181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:53.541028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":35,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:50.972914+0000","last_change":"2024-03-30T12:55:43.931142+0000","last_active":"2024-03-30T12:55:50.972914+0000","last_peered":"2024-03-30T12:55:50.972914+0000","last_clean":"2024-03-30T12:55:50.972914+0000","last_became_active":"2024-03-30T12:55:43.930722+0000","last_became_peered":"2024-03-30T12:55:43.930722+0000","last_unstale":"2024-03-30T12:55:50.972914+0000","last_undegraded":"2024-03-30T12:55:50.972914+0000","last_fullsized":"2024-03-30T12:55:50.972914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:47.641687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.192310+0000","last_change":"2024-03-30T12:55:43.933369+0000","last_active":"2024-03-30T12:55:52.192310+0000","last_peered":"2024-03-30T12:55:52.192310+0000","last_clean":"2024-03-30T12:55:52.192310+0000","last_became_active":"2024-03-30T12:55:43.932863+0000","last_became_peered":"2024-03-30T12:55:43.932863+0000","last_unstale":"2024-03-30T12:55:52.192310+0000","last_undegraded":"2024-03-30T12:55:52.192310+0000","last_fullsized":"2024-03-30T12:55:52.192310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:59.005903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:50.973469+0000","last_change":"2024-03-30T12:55:43.936909+0000","last_active":"2024-03-30T12:55:50.973469+0000","last_peered":"2024-03-30T12:55:50.973469+0000","last_clean":"2024-03-30T12:55:50.973469+0000","last_became_active":"2024-03-30T12:55:43.936547+0000","last_became_peered":"2024-03-30T12:55:43.936547+0000","last_unstale":"2024-03-30T12:55:50.973469+0000","last_undegraded":"2024-03-30T12:55:50.973469+0000","last_fullsized":"2024-03-30T12:55:50.973469+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:07.670156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":46,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.193092+0000","last_change":"2024-03-30T12:55:43.936225+0000","last_active":"2024-03-30T12:55:52.193092+0000","last_peered":"2024-03-30T12:55:52.193092+0000","last_clean":"2024-03-30T12:55:52.193092+0000","last_became_active":"2024-03-30T12:55:43.935038+0000","last_became_peered":"2024-03-30T12:55:43.935038+0000","last_unstale":"2024-03-30T12:55:52.193092+0000","last_undegraded":"2024-03-30T12:55:52.193092+0000","last_fullsized":"2024-03-30T12:55:52.193092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:53.215113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":54,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.193093+0000","last_change":"2024-03-30T12:55:43.931046+0000","last_active":"2024-03-30T12:55:52.193093+0000","last_peered":"2024-03-30T12:55:52.193093+0000","last_clean":"2024-03-30T12:55:52.193093+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T12:55:52.193093+0000","last_undegraded":"2024-03-30T12:55:52.193093+0000","last_fullsized":"2024-03-30T12:55:52.193093+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:09.603399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":39,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.951564+0000","last_change":"2024-03-30T12:55:43.936654+0000","last_active":"2024-03-30T12:55:51.951564+0000","last_peered":"2024-03-30T12:55:51.951564+0000","last_clean":"2024-03-30T12:55:51.951564+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T12:55:51.951564+0000","last_undegraded":"2024-03-30T12:55:51.951564+0000","last_fullsized":"2024-03-30T12:55:51.951564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:53.756917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.600158+0000","last_change":"2024-03-30T12:55:43.935508+0000","last_active":"2024-03-30T12:55:51.600158+0000","last_peered":"2024-03-30T12:55:51.600158+0000","last_clean":"2024-03-30T12:55:51.600158+0000","last_became_active":"2024-03-30T12:55:43.934495+0000","last_became_peered":"2024-03-30T12:55:43.934495+0000","last_unstale":"2024-03-30T12:55:51.600158+0000","last_undegraded":"2024-03-30T12:55:51.600158+0000","last_fullsized":"2024-03-30T12:55:51.600158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:48.367617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":39,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.951580+0000","last_change":"2024-03-30T12:55:43.935230+0000","last_active":"2024-03-30T12:55:51.951580+0000","last_peered":"2024-03-30T12:55:51.951580+0000","last_clean":"2024-03-30T12:55:51.951580+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T12:55:51.951580+0000","last_undegraded":"2024-03-30T12:55:51.951580+0000","last_fullsized":"2024-03-30T12:55:51.951580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:47.344018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:50.972853+0000","last_change":"2024-03-30T12:55:43.926406+0000","last_active":"2024-03-30T12:55:50.972853+0000","last_peered":"2024-03-30T12:55:50.972853+0000","last_clean":"2024-03-30T12:55:50.972853+0000","last_became_active":"2024-03-30T12:55:43.925945+0000","last_became_peered":"2024-03-30T12:55:43.925945+0000","last_unstale":"2024-03-30T12:55:50.972853+0000","last_undegraded":"2024-03-30T12:55:50.972853+0000","last_fullsized":"2024-03-30T12:55:50.972853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:08.300071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":103,"num_read_kb":4624,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94236672,"data_stored":94236672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509455,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42380,"kb_used_data":15360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329460,"statfs":{"total":96636764160,"available":96593367040,"internally_reserved":0,"allocated":15728640,"data_stored":15562249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542160,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45904,"kb_used_data":18884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94325936,"statfs":{"total":96636764160,"available":96589758464,"internally_reserved":0,"allocated":19337216,"data_stored":19170825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509456,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38644,"kb_used_data":11620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333196,"statfs":{"total":96636764160,"available":96597192704,"internally_reserved":0,"allocated":11898880,"data_stored":11732489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509456,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33596,"kb_used_data":6576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338244,"statfs":{"total":96636764160,"available":96602361856,"internally_reserved":0,"allocated":6733824,"data_stored":6567433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509456,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32444,"kb_used_data":5420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":5550080,"data_stored":5383689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509455,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40772,"kb_used_data":13752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331068,"statfs":{"total":96636764160,"available":96595013632,"internally_reserved":0,"allocated":14082048,"data_stored":13915657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542160,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36700,"kb_used_data":9680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335140,"statfs":{"total":96636764160,"available":96599183360,"internally_reserved":0,"allocated":9912320,"data_stored":9745929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574865,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39772,"kb_used_data":12752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332068,"statfs":{"total":96636764160,"available":96596037632,"internally_reserved":0,"allocated":13058048,"data_stored":12891657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11640832,"data_stored":11640832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19079168,"data_stored":19079168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9654272,"data_stored":9654272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:56:40.457 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:56:40.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:56:40.575+0000 7f567416a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:56:40.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:56:40.587+0000 7f567416a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:56:40.765 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:56:40.765 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T12:56:40.783 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-03-30T12:56:39.800209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":103,"num_read_kb":4624,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":310212,"kb_used_data":94044,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754664508,"statfs":{"total":773094113280,"available":772776456192,"internally_reserved":0,"allocated":96301056,"data_stored":94969928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004259"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.193120+0000","last_change":"2024-03-30T12:55:43.931163+0000","last_active":"2024-03-30T12:55:52.193120+0000","last_peered":"2024-03-30T12:55:52.193120+0000","last_clean":"2024-03-30T12:55:52.193120+0000","last_became_active":"2024-03-30T12:55:43.930823+0000","last_became_peered":"2024-03-30T12:55:43.930823+0000","last_unstale":"2024-03-30T12:55:52.193120+0000","last_undegraded":"2024-03-30T12:55:52.193120+0000","last_fullsized":"2024-03-30T12:55:52.193120+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:54.174180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:50.973407+0000","last_change":"2024-03-30T12:55:43.930916+0000","last_active":"2024-03-30T12:55:50.973407+0000","last_peered":"2024-03-30T12:55:50.973407+0000","last_clean":"2024-03-30T12:55:50.973407+0000","last_became_active":"2024-03-30T12:55:43.930591+0000","last_became_peered":"2024-03-30T12:55:43.930591+0000","last_unstale":"2024-03-30T12:55:50.973407+0000","last_undegraded":"2024-03-30T12:55:50.973407+0000","last_fullsized":"2024-03-30T12:55:50.973407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:07.237338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":34,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.600194+0000","last_change":"2024-03-30T12:55:43.931416+0000","last_active":"2024-03-30T12:55:51.600194+0000","last_peered":"2024-03-30T12:55:51.600194+0000","last_clean":"2024-03-30T12:55:51.600194+0000","last_became_active":"2024-03-30T12:55:43.931053+0000","last_became_peered":"2024-03-30T12:55:43.931053+0000","last_unstale":"2024-03-30T12:55:51.600194+0000","last_undegraded":"2024-03-30T12:55:51.600194+0000","last_fullsized":"2024-03-30T12:55:51.600194+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:02.687981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.600376+0000","last_change":"2024-03-30T12:55:43.938795+0000","last_active":"2024-03-30T12:55:51.600376+0000","last_peered":"2024-03-30T12:55:51.600376+0000","last_clean":"2024-03-30T12:55:51.600376+0000","last_became_active":"2024-03-30T12:55:43.938430+0000","last_became_peered":"2024-03-30T12:55:43.938430+0000","last_unstale":"2024-03-30T12:55:51.600376+0000","last_undegraded":"2024-03-30T12:55:51.600376+0000","last_fullsized":"2024-03-30T12:55:51.600376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:44.051242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.951550+0000","last_change":"2024-03-30T12:55:43.934224+0000","last_active":"2024-03-30T12:55:51.951550+0000","last_peered":"2024-03-30T12:55:51.951550+0000","last_clean":"2024-03-30T12:55:51.951550+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T12:55:51.951550+0000","last_undegraded":"2024-03-30T12:55:51.951550+0000","last_fullsized":"2024-03-30T12:55:51.951550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:03.609885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":42,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.192795+0000","last_change":"2024-03-30T12:55:43.927603+0000","last_active":"2024-03-30T12:55:52.192795+0000","last_peered":"2024-03-30T12:55:52.192795+0000","last_clean":"2024-03-30T12:55:52.192795+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T12:55:52.192795+0000","last_undegraded":"2024-03-30T12:55:52.192795+0000","last_fullsized":"2024-03-30T12:55:52.192795+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:47.393835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":46,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.600181+0000","last_change":"2024-03-30T12:55:43.934166+0000","last_active":"2024-03-30T12:55:51.600181+0000","last_peered":"2024-03-30T12:55:51.600181+0000","last_clean":"2024-03-30T12:55:51.600181+0000","last_became_active":"2024-03-30T12:55:43.933833+0000","last_became_peered":"2024-03-30T12:55:43.933833+0000","last_unstale":"2024-03-30T12:55:51.600181+0000","last_undegraded":"2024-03-30T12:55:51.600181+0000","last_fullsized":"2024-03-30T12:55:51.600181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:53.541028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":35,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:50.972914+0000","last_change":"2024-03-30T12:55:43.931142+0000","last_active":"2024-03-30T12:55:50.972914+0000","last_peered":"2024-03-30T12:55:50.972914+0000","last_clean":"2024-03-30T12:55:50.972914+0000","last_became_active":"2024-03-30T12:55:43.930722+0000","last_became_peered":"2024-03-30T12:55:43.930722+0000","last_unstale":"2024-03-30T12:55:50.972914+0000","last_undegraded":"2024-03-30T12:55:50.972914+0000","last_fullsized":"2024-03-30T12:55:50.972914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:47.641687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.192310+0000","last_change":"2024-03-30T12:55:43.933369+0000","last_active":"2024-03-30T12:55:52.192310+0000","last_peered":"2024-03-30T12:55:52.192310+0000","last_clean":"2024-03-30T12:55:52.192310+0000","last_became_active":"2024-03-30T12:55:43.932863+0000","last_became_peered":"2024-03-30T12:55:43.932863+0000","last_unstale":"2024-03-30T12:55:52.192310+0000","last_undegraded":"2024-03-30T12:55:52.192310+0000","last_fullsized":"2024-03-30T12:55:52.192310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:59.005903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:50.973469+0000","last_change":"2024-03-30T12:55:43.936909+0000","last_active":"2024-03-30T12:55:50.973469+0000","last_peered":"2024-03-30T12:55:50.973469+0000","last_clean":"2024-03-30T12:55:50.973469+0000","last_became_active":"2024-03-30T12:55:43.936547+0000","last_became_peered":"2024-03-30T12:55:43.936547+0000","last_unstale":"2024-03-30T12:55:50.973469+0000","last_undegraded":"2024-03-30T12:55:50.973469+0000","last_fullsized":"2024-03-30T12:55:50.973469+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:07.670156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":46,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.193092+0000","last_change":"2024-03-30T12:55:43.936225+0000","last_active":"2024-03-30T12:55:52.193092+0000","last_peered":"2024-03-30T12:55:52.193092+0000","last_clean":"2024-03-30T12:55:52.193092+0000","last_became_active":"2024-03-30T12:55:43.935038+0000","last_became_peered":"2024-03-30T12:55:43.935038+0000","last_unstale":"2024-03-30T12:55:52.193092+0000","last_undegraded":"2024-03-30T12:55:52.193092+0000","last_fullsized":"2024-03-30T12:55:52.193092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:53.215113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":54,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.193093+0000","last_change":"2024-03-30T12:55:43.931046+0000","last_active":"2024-03-30T12:55:52.193093+0000","last_peered":"2024-03-30T12:55:52.193093+0000","last_clean":"2024-03-30T12:55:52.193093+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T12:55:52.193093+0000","last_undegraded":"2024-03-30T12:55:52.193093+0000","last_fullsized":"2024-03-30T12:55:52.193093+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:09.603399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":39,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.951564+0000","last_change":"2024-03-30T12:55:43.936654+0000","last_active":"2024-03-30T12:55:51.951564+0000","last_peered":"2024-03-30T12:55:51.951564+0000","last_clean":"2024-03-30T12:55:51.951564+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T12:55:51.951564+0000","last_undegraded":"2024-03-30T12:55:51.951564+0000","last_fullsized":"2024-03-30T12:55:51.951564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:53.756917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.600158+0000","last_change":"2024-03-30T12:55:43.935508+0000","last_active":"2024-03-30T12:55:51.600158+0000","last_peered":"2024-03-30T12:55:51.600158+0000","last_clean":"2024-03-30T12:55:51.600158+0000","last_became_active":"2024-03-30T12:55:43.934495+0000","last_became_peered":"2024-03-30T12:55:43.934495+0000","last_unstale":"2024-03-30T12:55:51.600158+0000","last_undegraded":"2024-03-30T12:55:51.600158+0000","last_fullsized":"2024-03-30T12:55:51.600158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:48.367617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":39,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.951580+0000","last_change":"2024-03-30T12:55:43.935230+0000","last_active":"2024-03-30T12:55:51.951580+0000","last_peered":"2024-03-30T12:55:51.951580+0000","last_clean":"2024-03-30T12:55:51.951580+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T12:55:51.951580+0000","last_undegraded":"2024-03-30T12:55:51.951580+0000","last_fullsized":"2024-03-30T12:55:51.951580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:47.344018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:50.972853+0000","last_change":"2024-03-30T12:55:43.926406+0000","last_active":"2024-03-30T12:55:50.972853+0000","last_peered":"2024-03-30T12:55:50.972853+0000","last_clean":"2024-03-30T12:55:50.972853+0000","last_became_active":"2024-03-30T12:55:43.925945+0000","last_became_peered":"2024-03-30T12:55:43.925945+0000","last_unstale":"2024-03-30T12:55:50.972853+0000","last_undegraded":"2024-03-30T12:55:50.972853+0000","last_fullsized":"2024-03-30T12:55:50.972853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:08.300071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":103,"num_read_kb":4624,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94236672,"data_stored":94236672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509455,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42380,"kb_used_data":15360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329460,"statfs":{"total":96636764160,"available":96593367040,"internally_reserved":0,"allocated":15728640,"data_stored":15562249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542160,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45904,"kb_used_data":18884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94325936,"statfs":{"total":96636764160,"available":96589758464,"internally_reserved":0,"allocated":19337216,"data_stored":19170825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509456,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38644,"kb_used_data":11620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333196,"statfs":{"total":96636764160,"available":96597192704,"internally_reserved":0,"allocated":11898880,"data_stored":11732489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509456,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33596,"kb_used_data":6576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338244,"statfs":{"total":96636764160,"available":96602361856,"internally_reserved":0,"allocated":6733824,"data_stored":6567433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509456,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32444,"kb_used_data":5420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":5550080,"data_stored":5383689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509455,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40772,"kb_used_data":13752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331068,"statfs":{"total":96636764160,"available":96595013632,"internally_reserved":0,"allocated":14082048,"data_stored":13915657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542160,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36700,"kb_used_data":9680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335140,"statfs":{"total":96636764160,"available":96599183360,"internally_reserved":0,"allocated":9912320,"data_stored":9745929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574865,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39772,"kb_used_data":12752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332068,"statfs":{"total":96636764160,"available":96596037632,"internally_reserved":0,"allocated":13058048,"data_stored":12891657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11640832,"data_stored":11640832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19079168,"data_stored":19079168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9654272,"data_stored":9654272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:56:40.784 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T12:56:40.784 INFO:tasks.thrashosds.thrasher:doing min_size thrashing 2024-03-30T12:56:40.784 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-30T12:56:40.784 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:56:40.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:56:40.903+0000 7f3d43e50640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:56:40.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:56:40.915+0000 7f3d43e50640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:56:41.095 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:56:41.095 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T12:56:41.114 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-03-30T12:56:39.800209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":103,"num_read_kb":4624,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":310212,"kb_used_data":94044,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754664508,"statfs":{"total":773094113280,"available":772776456192,"internally_reserved":0,"allocated":96301056,"data_stored":94969928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004259"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.193120+0000","last_change":"2024-03-30T12:55:43.931163+0000","last_active":"2024-03-30T12:55:52.193120+0000","last_peered":"2024-03-30T12:55:52.193120+0000","last_clean":"2024-03-30T12:55:52.193120+0000","last_became_active":"2024-03-30T12:55:43.930823+0000","last_became_peered":"2024-03-30T12:55:43.930823+0000","last_unstale":"2024-03-30T12:55:52.193120+0000","last_undegraded":"2024-03-30T12:55:52.193120+0000","last_fullsized":"2024-03-30T12:55:52.193120+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:54.174180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:50.973407+0000","last_change":"2024-03-30T12:55:43.930916+0000","last_active":"2024-03-30T12:55:50.973407+0000","last_peered":"2024-03-30T12:55:50.973407+0000","last_clean":"2024-03-30T12:55:50.973407+0000","last_became_active":"2024-03-30T12:55:43.930591+0000","last_became_peered":"2024-03-30T12:55:43.930591+0000","last_unstale":"2024-03-30T12:55:50.973407+0000","last_undegraded":"2024-03-30T12:55:50.973407+0000","last_fullsized":"2024-03-30T12:55:50.973407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:07.237338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":34,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.600194+0000","last_change":"2024-03-30T12:55:43.931416+0000","last_active":"2024-03-30T12:55:51.600194+0000","last_peered":"2024-03-30T12:55:51.600194+0000","last_clean":"2024-03-30T12:55:51.600194+0000","last_became_active":"2024-03-30T12:55:43.931053+0000","last_became_peered":"2024-03-30T12:55:43.931053+0000","last_unstale":"2024-03-30T12:55:51.600194+0000","last_undegraded":"2024-03-30T12:55:51.600194+0000","last_fullsized":"2024-03-30T12:55:51.600194+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:02.687981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.600376+0000","last_change":"2024-03-30T12:55:43.938795+0000","last_active":"2024-03-30T12:55:51.600376+0000","last_peered":"2024-03-30T12:55:51.600376+0000","last_clean":"2024-03-30T12:55:51.600376+0000","last_became_active":"2024-03-30T12:55:43.938430+0000","last_became_peered":"2024-03-30T12:55:43.938430+0000","last_unstale":"2024-03-30T12:55:51.600376+0000","last_undegraded":"2024-03-30T12:55:51.600376+0000","last_fullsized":"2024-03-30T12:55:51.600376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:44.051242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.951550+0000","last_change":"2024-03-30T12:55:43.934224+0000","last_active":"2024-03-30T12:55:51.951550+0000","last_peered":"2024-03-30T12:55:51.951550+0000","last_clean":"2024-03-30T12:55:51.951550+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T12:55:51.951550+0000","last_undegraded":"2024-03-30T12:55:51.951550+0000","last_fullsized":"2024-03-30T12:55:51.951550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:03.609885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":42,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.192795+0000","last_change":"2024-03-30T12:55:43.927603+0000","last_active":"2024-03-30T12:55:52.192795+0000","last_peered":"2024-03-30T12:55:52.192795+0000","last_clean":"2024-03-30T12:55:52.192795+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T12:55:52.192795+0000","last_undegraded":"2024-03-30T12:55:52.192795+0000","last_fullsized":"2024-03-30T12:55:52.192795+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:47.393835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":46,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.600181+0000","last_change":"2024-03-30T12:55:43.934166+0000","last_active":"2024-03-30T12:55:51.600181+0000","last_peered":"2024-03-30T12:55:51.600181+0000","last_clean":"2024-03-30T12:55:51.600181+0000","last_became_active":"2024-03-30T12:55:43.933833+0000","last_became_peered":"2024-03-30T12:55:43.933833+0000","last_unstale":"2024-03-30T12:55:51.600181+0000","last_undegraded":"2024-03-30T12:55:51.600181+0000","last_fullsized":"2024-03-30T12:55:51.600181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:53.541028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":35,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:50.972914+0000","last_change":"2024-03-30T12:55:43.931142+0000","last_active":"2024-03-30T12:55:50.972914+0000","last_peered":"2024-03-30T12:55:50.972914+0000","last_clean":"2024-03-30T12:55:50.972914+0000","last_became_active":"2024-03-30T12:55:43.930722+0000","last_became_peered":"2024-03-30T12:55:43.930722+0000","last_unstale":"2024-03-30T12:55:50.972914+0000","last_undegraded":"2024-03-30T12:55:50.972914+0000","last_fullsized":"2024-03-30T12:55:50.972914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:47.641687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.192310+0000","last_change":"2024-03-30T12:55:43.933369+0000","last_active":"2024-03-30T12:55:52.192310+0000","last_peered":"2024-03-30T12:55:52.192310+0000","last_clean":"2024-03-30T12:55:52.192310+0000","last_became_active":"2024-03-30T12:55:43.932863+0000","last_became_peered":"2024-03-30T12:55:43.932863+0000","last_unstale":"2024-03-30T12:55:52.192310+0000","last_undegraded":"2024-03-30T12:55:52.192310+0000","last_fullsized":"2024-03-30T12:55:52.192310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:59.005903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:50.973469+0000","last_change":"2024-03-30T12:55:43.936909+0000","last_active":"2024-03-30T12:55:50.973469+0000","last_peered":"2024-03-30T12:55:50.973469+0000","last_clean":"2024-03-30T12:55:50.973469+0000","last_became_active":"2024-03-30T12:55:43.936547+0000","last_became_peered":"2024-03-30T12:55:43.936547+0000","last_unstale":"2024-03-30T12:55:50.973469+0000","last_undegraded":"2024-03-30T12:55:50.973469+0000","last_fullsized":"2024-03-30T12:55:50.973469+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:07.670156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":46,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.193092+0000","last_change":"2024-03-30T12:55:43.936225+0000","last_active":"2024-03-30T12:55:52.193092+0000","last_peered":"2024-03-30T12:55:52.193092+0000","last_clean":"2024-03-30T12:55:52.193092+0000","last_became_active":"2024-03-30T12:55:43.935038+0000","last_became_peered":"2024-03-30T12:55:43.935038+0000","last_unstale":"2024-03-30T12:55:52.193092+0000","last_undegraded":"2024-03-30T12:55:52.193092+0000","last_fullsized":"2024-03-30T12:55:52.193092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:53.215113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":54,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.193093+0000","last_change":"2024-03-30T12:55:43.931046+0000","last_active":"2024-03-30T12:55:52.193093+0000","last_peered":"2024-03-30T12:55:52.193093+0000","last_clean":"2024-03-30T12:55:52.193093+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T12:55:52.193093+0000","last_undegraded":"2024-03-30T12:55:52.193093+0000","last_fullsized":"2024-03-30T12:55:52.193093+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:09.603399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":39,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.951564+0000","last_change":"2024-03-30T12:55:43.936654+0000","last_active":"2024-03-30T12:55:51.951564+0000","last_peered":"2024-03-30T12:55:51.951564+0000","last_clean":"2024-03-30T12:55:51.951564+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T12:55:51.951564+0000","last_undegraded":"2024-03-30T12:55:51.951564+0000","last_fullsized":"2024-03-30T12:55:51.951564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:53.756917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.600158+0000","last_change":"2024-03-30T12:55:43.935508+0000","last_active":"2024-03-30T12:55:51.600158+0000","last_peered":"2024-03-30T12:55:51.600158+0000","last_clean":"2024-03-30T12:55:51.600158+0000","last_became_active":"2024-03-30T12:55:43.934495+0000","last_became_peered":"2024-03-30T12:55:43.934495+0000","last_unstale":"2024-03-30T12:55:51.600158+0000","last_undegraded":"2024-03-30T12:55:51.600158+0000","last_fullsized":"2024-03-30T12:55:51.600158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:48.367617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":39,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.951580+0000","last_change":"2024-03-30T12:55:43.935230+0000","last_active":"2024-03-30T12:55:51.951580+0000","last_peered":"2024-03-30T12:55:51.951580+0000","last_clean":"2024-03-30T12:55:51.951580+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T12:55:51.951580+0000","last_undegraded":"2024-03-30T12:55:51.951580+0000","last_fullsized":"2024-03-30T12:55:51.951580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:47.344018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:50.972853+0000","last_change":"2024-03-30T12:55:43.926406+0000","last_active":"2024-03-30T12:55:50.972853+0000","last_peered":"2024-03-30T12:55:50.972853+0000","last_clean":"2024-03-30T12:55:50.972853+0000","last_became_active":"2024-03-30T12:55:43.925945+0000","last_became_peered":"2024-03-30T12:55:43.925945+0000","last_unstale":"2024-03-30T12:55:50.972853+0000","last_undegraded":"2024-03-30T12:55:50.972853+0000","last_fullsized":"2024-03-30T12:55:50.972853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:08.300071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":103,"num_read_kb":4624,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94236672,"data_stored":94236672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509455,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42380,"kb_used_data":15360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329460,"statfs":{"total":96636764160,"available":96593367040,"internally_reserved":0,"allocated":15728640,"data_stored":15562249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542160,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45904,"kb_used_data":18884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94325936,"statfs":{"total":96636764160,"available":96589758464,"internally_reserved":0,"allocated":19337216,"data_stored":19170825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509456,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38644,"kb_used_data":11620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333196,"statfs":{"total":96636764160,"available":96597192704,"internally_reserved":0,"allocated":11898880,"data_stored":11732489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509456,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33596,"kb_used_data":6576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338244,"statfs":{"total":96636764160,"available":96602361856,"internally_reserved":0,"allocated":6733824,"data_stored":6567433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509456,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32444,"kb_used_data":5420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":5550080,"data_stored":5383689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509455,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40772,"kb_used_data":13752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331068,"statfs":{"total":96636764160,"available":96595013632,"internally_reserved":0,"allocated":14082048,"data_stored":13915657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542160,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36700,"kb_used_data":9680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335140,"statfs":{"total":96636764160,"available":96599183360,"internally_reserved":0,"allocated":9912320,"data_stored":9745929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574865,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39772,"kb_used_data":12752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332068,"statfs":{"total":96636764160,"available":96596037632,"internally_reserved":0,"allocated":13058048,"data_stored":12891657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11640832,"data_stored":11640832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19079168,"data_stored":19079168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9654272,"data_stored":9654272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:56:41.115 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:56:41.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:56:41.223+0000 7fe9a4b67640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:56:41.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:56:41.235+0000 7fe9a4b67640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:56:41.406 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:56:41.407 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T12:56:41.423 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-03-30T12:56:39.800209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":103,"num_read_kb":4624,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":310212,"kb_used_data":94044,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754664508,"statfs":{"total":773094113280,"available":772776456192,"internally_reserved":0,"allocated":96301056,"data_stored":94969928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004259"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.193120+0000","last_change":"2024-03-30T12:55:43.931163+0000","last_active":"2024-03-30T12:55:52.193120+0000","last_peered":"2024-03-30T12:55:52.193120+0000","last_clean":"2024-03-30T12:55:52.193120+0000","last_became_active":"2024-03-30T12:55:43.930823+0000","last_became_peered":"2024-03-30T12:55:43.930823+0000","last_unstale":"2024-03-30T12:55:52.193120+0000","last_undegraded":"2024-03-30T12:55:52.193120+0000","last_fullsized":"2024-03-30T12:55:52.193120+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:54.174180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:50.973407+0000","last_change":"2024-03-30T12:55:43.930916+0000","last_active":"2024-03-30T12:55:50.973407+0000","last_peered":"2024-03-30T12:55:50.973407+0000","last_clean":"2024-03-30T12:55:50.973407+0000","last_became_active":"2024-03-30T12:55:43.930591+0000","last_became_peered":"2024-03-30T12:55:43.930591+0000","last_unstale":"2024-03-30T12:55:50.973407+0000","last_undegraded":"2024-03-30T12:55:50.973407+0000","last_fullsized":"2024-03-30T12:55:50.973407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:07.237338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":34,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.600194+0000","last_change":"2024-03-30T12:55:43.931416+0000","last_active":"2024-03-30T12:55:51.600194+0000","last_peered":"2024-03-30T12:55:51.600194+0000","last_clean":"2024-03-30T12:55:51.600194+0000","last_became_active":"2024-03-30T12:55:43.931053+0000","last_became_peered":"2024-03-30T12:55:43.931053+0000","last_unstale":"2024-03-30T12:55:51.600194+0000","last_undegraded":"2024-03-30T12:55:51.600194+0000","last_fullsized":"2024-03-30T12:55:51.600194+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:02.687981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.600376+0000","last_change":"2024-03-30T12:55:43.938795+0000","last_active":"2024-03-30T12:55:51.600376+0000","last_peered":"2024-03-30T12:55:51.600376+0000","last_clean":"2024-03-30T12:55:51.600376+0000","last_became_active":"2024-03-30T12:55:43.938430+0000","last_became_peered":"2024-03-30T12:55:43.938430+0000","last_unstale":"2024-03-30T12:55:51.600376+0000","last_undegraded":"2024-03-30T12:55:51.600376+0000","last_fullsized":"2024-03-30T12:55:51.600376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:44.051242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.951550+0000","last_change":"2024-03-30T12:55:43.934224+0000","last_active":"2024-03-30T12:55:51.951550+0000","last_peered":"2024-03-30T12:55:51.951550+0000","last_clean":"2024-03-30T12:55:51.951550+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T12:55:51.951550+0000","last_undegraded":"2024-03-30T12:55:51.951550+0000","last_fullsized":"2024-03-30T12:55:51.951550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:03.609885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":42,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.192795+0000","last_change":"2024-03-30T12:55:43.927603+0000","last_active":"2024-03-30T12:55:52.192795+0000","last_peered":"2024-03-30T12:55:52.192795+0000","last_clean":"2024-03-30T12:55:52.192795+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T12:55:52.192795+0000","last_undegraded":"2024-03-30T12:55:52.192795+0000","last_fullsized":"2024-03-30T12:55:52.192795+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:47.393835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":46,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.600181+0000","last_change":"2024-03-30T12:55:43.934166+0000","last_active":"2024-03-30T12:55:51.600181+0000","last_peered":"2024-03-30T12:55:51.600181+0000","last_clean":"2024-03-30T12:55:51.600181+0000","last_became_active":"2024-03-30T12:55:43.933833+0000","last_became_peered":"2024-03-30T12:55:43.933833+0000","last_unstale":"2024-03-30T12:55:51.600181+0000","last_undegraded":"2024-03-30T12:55:51.600181+0000","last_fullsized":"2024-03-30T12:55:51.600181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:53.541028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":35,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:50.972914+0000","last_change":"2024-03-30T12:55:43.931142+0000","last_active":"2024-03-30T12:55:50.972914+0000","last_peered":"2024-03-30T12:55:50.972914+0000","last_clean":"2024-03-30T12:55:50.972914+0000","last_became_active":"2024-03-30T12:55:43.930722+0000","last_became_peered":"2024-03-30T12:55:43.930722+0000","last_unstale":"2024-03-30T12:55:50.972914+0000","last_undegraded":"2024-03-30T12:55:50.972914+0000","last_fullsized":"2024-03-30T12:55:50.972914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:47.641687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.192310+0000","last_change":"2024-03-30T12:55:43.933369+0000","last_active":"2024-03-30T12:55:52.192310+0000","last_peered":"2024-03-30T12:55:52.192310+0000","last_clean":"2024-03-30T12:55:52.192310+0000","last_became_active":"2024-03-30T12:55:43.932863+0000","last_became_peered":"2024-03-30T12:55:43.932863+0000","last_unstale":"2024-03-30T12:55:52.192310+0000","last_undegraded":"2024-03-30T12:55:52.192310+0000","last_fullsized":"2024-03-30T12:55:52.192310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:59.005903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:50.973469+0000","last_change":"2024-03-30T12:55:43.936909+0000","last_active":"2024-03-30T12:55:50.973469+0000","last_peered":"2024-03-30T12:55:50.973469+0000","last_clean":"2024-03-30T12:55:50.973469+0000","last_became_active":"2024-03-30T12:55:43.936547+0000","last_became_peered":"2024-03-30T12:55:43.936547+0000","last_unstale":"2024-03-30T12:55:50.973469+0000","last_undegraded":"2024-03-30T12:55:50.973469+0000","last_fullsized":"2024-03-30T12:55:50.973469+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:07.670156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":46,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.193092+0000","last_change":"2024-03-30T12:55:43.936225+0000","last_active":"2024-03-30T12:55:52.193092+0000","last_peered":"2024-03-30T12:55:52.193092+0000","last_clean":"2024-03-30T12:55:52.193092+0000","last_became_active":"2024-03-30T12:55:43.935038+0000","last_became_peered":"2024-03-30T12:55:43.935038+0000","last_unstale":"2024-03-30T12:55:52.193092+0000","last_undegraded":"2024-03-30T12:55:52.193092+0000","last_fullsized":"2024-03-30T12:55:52.193092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:53.215113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":54,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.193093+0000","last_change":"2024-03-30T12:55:43.931046+0000","last_active":"2024-03-30T12:55:52.193093+0000","last_peered":"2024-03-30T12:55:52.193093+0000","last_clean":"2024-03-30T12:55:52.193093+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T12:55:52.193093+0000","last_undegraded":"2024-03-30T12:55:52.193093+0000","last_fullsized":"2024-03-30T12:55:52.193093+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:09.603399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":39,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.951564+0000","last_change":"2024-03-30T12:55:43.936654+0000","last_active":"2024-03-30T12:55:51.951564+0000","last_peered":"2024-03-30T12:55:51.951564+0000","last_clean":"2024-03-30T12:55:51.951564+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T12:55:51.951564+0000","last_undegraded":"2024-03-30T12:55:51.951564+0000","last_fullsized":"2024-03-30T12:55:51.951564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:53.756917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.600158+0000","last_change":"2024-03-30T12:55:43.935508+0000","last_active":"2024-03-30T12:55:51.600158+0000","last_peered":"2024-03-30T12:55:51.600158+0000","last_clean":"2024-03-30T12:55:51.600158+0000","last_became_active":"2024-03-30T12:55:43.934495+0000","last_became_peered":"2024-03-30T12:55:43.934495+0000","last_unstale":"2024-03-30T12:55:51.600158+0000","last_undegraded":"2024-03-30T12:55:51.600158+0000","last_fullsized":"2024-03-30T12:55:51.600158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:48.367617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":39,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.951580+0000","last_change":"2024-03-30T12:55:43.935230+0000","last_active":"2024-03-30T12:55:51.951580+0000","last_peered":"2024-03-30T12:55:51.951580+0000","last_clean":"2024-03-30T12:55:51.951580+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T12:55:51.951580+0000","last_undegraded":"2024-03-30T12:55:51.951580+0000","last_fullsized":"2024-03-30T12:55:51.951580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:47.344018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:50.972853+0000","last_change":"2024-03-30T12:55:43.926406+0000","last_active":"2024-03-30T12:55:50.972853+0000","last_peered":"2024-03-30T12:55:50.972853+0000","last_clean":"2024-03-30T12:55:50.972853+0000","last_became_active":"2024-03-30T12:55:43.925945+0000","last_became_peered":"2024-03-30T12:55:43.925945+0000","last_unstale":"2024-03-30T12:55:50.972853+0000","last_undegraded":"2024-03-30T12:55:50.972853+0000","last_fullsized":"2024-03-30T12:55:50.972853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:08.300071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":103,"num_read_kb":4624,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94236672,"data_stored":94236672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509455,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42380,"kb_used_data":15360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329460,"statfs":{"total":96636764160,"available":96593367040,"internally_reserved":0,"allocated":15728640,"data_stored":15562249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542160,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45904,"kb_used_data":18884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94325936,"statfs":{"total":96636764160,"available":96589758464,"internally_reserved":0,"allocated":19337216,"data_stored":19170825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509456,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38644,"kb_used_data":11620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333196,"statfs":{"total":96636764160,"available":96597192704,"internally_reserved":0,"allocated":11898880,"data_stored":11732489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509456,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33596,"kb_used_data":6576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338244,"statfs":{"total":96636764160,"available":96602361856,"internally_reserved":0,"allocated":6733824,"data_stored":6567433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509456,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32444,"kb_used_data":5420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":5550080,"data_stored":5383689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509455,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40772,"kb_used_data":13752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331068,"statfs":{"total":96636764160,"available":96595013632,"internally_reserved":0,"allocated":14082048,"data_stored":13915657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542160,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36700,"kb_used_data":9680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335140,"statfs":{"total":96636764160,"available":96599183360,"internally_reserved":0,"allocated":9912320,"data_stored":9745929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574865,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39772,"kb_used_data":12752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332068,"statfs":{"total":96636764160,"available":96596037632,"internally_reserved":0,"allocated":13058048,"data_stored":12891657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11640832,"data_stored":11640832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19079168,"data_stored":19079168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9654272,"data_stored":9654272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:56:41.424 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-30T12:56:41.424 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:56:41.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:56:41.523+0000 7fb608c69640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:56:41.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:56:41.539+0000 7fb608c69640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:56:41.706 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:56:41.706 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T12:56:41.723 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-03-30T12:56:39.800209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":103,"num_read_kb":4624,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":310212,"kb_used_data":94044,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754664508,"statfs":{"total":773094113280,"available":772776456192,"internally_reserved":0,"allocated":96301056,"data_stored":94969928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004259"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.193120+0000","last_change":"2024-03-30T12:55:43.931163+0000","last_active":"2024-03-30T12:55:52.193120+0000","last_peered":"2024-03-30T12:55:52.193120+0000","last_clean":"2024-03-30T12:55:52.193120+0000","last_became_active":"2024-03-30T12:55:43.930823+0000","last_became_peered":"2024-03-30T12:55:43.930823+0000","last_unstale":"2024-03-30T12:55:52.193120+0000","last_undegraded":"2024-03-30T12:55:52.193120+0000","last_fullsized":"2024-03-30T12:55:52.193120+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:54.174180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:50.973407+0000","last_change":"2024-03-30T12:55:43.930916+0000","last_active":"2024-03-30T12:55:50.973407+0000","last_peered":"2024-03-30T12:55:50.973407+0000","last_clean":"2024-03-30T12:55:50.973407+0000","last_became_active":"2024-03-30T12:55:43.930591+0000","last_became_peered":"2024-03-30T12:55:43.930591+0000","last_unstale":"2024-03-30T12:55:50.973407+0000","last_undegraded":"2024-03-30T12:55:50.973407+0000","last_fullsized":"2024-03-30T12:55:50.973407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:07.237338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":34,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.600194+0000","last_change":"2024-03-30T12:55:43.931416+0000","last_active":"2024-03-30T12:55:51.600194+0000","last_peered":"2024-03-30T12:55:51.600194+0000","last_clean":"2024-03-30T12:55:51.600194+0000","last_became_active":"2024-03-30T12:55:43.931053+0000","last_became_peered":"2024-03-30T12:55:43.931053+0000","last_unstale":"2024-03-30T12:55:51.600194+0000","last_undegraded":"2024-03-30T12:55:51.600194+0000","last_fullsized":"2024-03-30T12:55:51.600194+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:02.687981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.600376+0000","last_change":"2024-03-30T12:55:43.938795+0000","last_active":"2024-03-30T12:55:51.600376+0000","last_peered":"2024-03-30T12:55:51.600376+0000","last_clean":"2024-03-30T12:55:51.600376+0000","last_became_active":"2024-03-30T12:55:43.938430+0000","last_became_peered":"2024-03-30T12:55:43.938430+0000","last_unstale":"2024-03-30T12:55:51.600376+0000","last_undegraded":"2024-03-30T12:55:51.600376+0000","last_fullsized":"2024-03-30T12:55:51.600376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:44.051242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.951550+0000","last_change":"2024-03-30T12:55:43.934224+0000","last_active":"2024-03-30T12:55:51.951550+0000","last_peered":"2024-03-30T12:55:51.951550+0000","last_clean":"2024-03-30T12:55:51.951550+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T12:55:51.951550+0000","last_undegraded":"2024-03-30T12:55:51.951550+0000","last_fullsized":"2024-03-30T12:55:51.951550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:03.609885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":42,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.192795+0000","last_change":"2024-03-30T12:55:43.927603+0000","last_active":"2024-03-30T12:55:52.192795+0000","last_peered":"2024-03-30T12:55:52.192795+0000","last_clean":"2024-03-30T12:55:52.192795+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T12:55:52.192795+0000","last_undegraded":"2024-03-30T12:55:52.192795+0000","last_fullsized":"2024-03-30T12:55:52.192795+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:47.393835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":46,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.600181+0000","last_change":"2024-03-30T12:55:43.934166+0000","last_active":"2024-03-30T12:55:51.600181+0000","last_peered":"2024-03-30T12:55:51.600181+0000","last_clean":"2024-03-30T12:55:51.600181+0000","last_became_active":"2024-03-30T12:55:43.933833+0000","last_became_peered":"2024-03-30T12:55:43.933833+0000","last_unstale":"2024-03-30T12:55:51.600181+0000","last_undegraded":"2024-03-30T12:55:51.600181+0000","last_fullsized":"2024-03-30T12:55:51.600181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:53.541028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":35,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:50.972914+0000","last_change":"2024-03-30T12:55:43.931142+0000","last_active":"2024-03-30T12:55:50.972914+0000","last_peered":"2024-03-30T12:55:50.972914+0000","last_clean":"2024-03-30T12:55:50.972914+0000","last_became_active":"2024-03-30T12:55:43.930722+0000","last_became_peered":"2024-03-30T12:55:43.930722+0000","last_unstale":"2024-03-30T12:55:50.972914+0000","last_undegraded":"2024-03-30T12:55:50.972914+0000","last_fullsized":"2024-03-30T12:55:50.972914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:47.641687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.192310+0000","last_change":"2024-03-30T12:55:43.933369+0000","last_active":"2024-03-30T12:55:52.192310+0000","last_peered":"2024-03-30T12:55:52.192310+0000","last_clean":"2024-03-30T12:55:52.192310+0000","last_became_active":"2024-03-30T12:55:43.932863+0000","last_became_peered":"2024-03-30T12:55:43.932863+0000","last_unstale":"2024-03-30T12:55:52.192310+0000","last_undegraded":"2024-03-30T12:55:52.192310+0000","last_fullsized":"2024-03-30T12:55:52.192310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:59.005903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:50.973469+0000","last_change":"2024-03-30T12:55:43.936909+0000","last_active":"2024-03-30T12:55:50.973469+0000","last_peered":"2024-03-30T12:55:50.973469+0000","last_clean":"2024-03-30T12:55:50.973469+0000","last_became_active":"2024-03-30T12:55:43.936547+0000","last_became_peered":"2024-03-30T12:55:43.936547+0000","last_unstale":"2024-03-30T12:55:50.973469+0000","last_undegraded":"2024-03-30T12:55:50.973469+0000","last_fullsized":"2024-03-30T12:55:50.973469+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:07.670156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":46,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.193092+0000","last_change":"2024-03-30T12:55:43.936225+0000","last_active":"2024-03-30T12:55:52.193092+0000","last_peered":"2024-03-30T12:55:52.193092+0000","last_clean":"2024-03-30T12:55:52.193092+0000","last_became_active":"2024-03-30T12:55:43.935038+0000","last_became_peered":"2024-03-30T12:55:43.935038+0000","last_unstale":"2024-03-30T12:55:52.193092+0000","last_undegraded":"2024-03-30T12:55:52.193092+0000","last_fullsized":"2024-03-30T12:55:52.193092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:53.215113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":54,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.193093+0000","last_change":"2024-03-30T12:55:43.931046+0000","last_active":"2024-03-30T12:55:52.193093+0000","last_peered":"2024-03-30T12:55:52.193093+0000","last_clean":"2024-03-30T12:55:52.193093+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T12:55:52.193093+0000","last_undegraded":"2024-03-30T12:55:52.193093+0000","last_fullsized":"2024-03-30T12:55:52.193093+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:09.603399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":39,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.951564+0000","last_change":"2024-03-30T12:55:43.936654+0000","last_active":"2024-03-30T12:55:51.951564+0000","last_peered":"2024-03-30T12:55:51.951564+0000","last_clean":"2024-03-30T12:55:51.951564+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T12:55:51.951564+0000","last_undegraded":"2024-03-30T12:55:51.951564+0000","last_fullsized":"2024-03-30T12:55:51.951564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:53.756917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.600158+0000","last_change":"2024-03-30T12:55:43.935508+0000","last_active":"2024-03-30T12:55:51.600158+0000","last_peered":"2024-03-30T12:55:51.600158+0000","last_clean":"2024-03-30T12:55:51.600158+0000","last_became_active":"2024-03-30T12:55:43.934495+0000","last_became_peered":"2024-03-30T12:55:43.934495+0000","last_unstale":"2024-03-30T12:55:51.600158+0000","last_undegraded":"2024-03-30T12:55:51.600158+0000","last_fullsized":"2024-03-30T12:55:51.600158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:48.367617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":39,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.951580+0000","last_change":"2024-03-30T12:55:43.935230+0000","last_active":"2024-03-30T12:55:51.951580+0000","last_peered":"2024-03-30T12:55:51.951580+0000","last_clean":"2024-03-30T12:55:51.951580+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T12:55:51.951580+0000","last_undegraded":"2024-03-30T12:55:51.951580+0000","last_fullsized":"2024-03-30T12:55:51.951580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:47.344018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:50.972853+0000","last_change":"2024-03-30T12:55:43.926406+0000","last_active":"2024-03-30T12:55:50.972853+0000","last_peered":"2024-03-30T12:55:50.972853+0000","last_clean":"2024-03-30T12:55:50.972853+0000","last_became_active":"2024-03-30T12:55:43.925945+0000","last_became_peered":"2024-03-30T12:55:43.925945+0000","last_unstale":"2024-03-30T12:55:50.972853+0000","last_undegraded":"2024-03-30T12:55:50.972853+0000","last_fullsized":"2024-03-30T12:55:50.972853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:08.300071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":103,"num_read_kb":4624,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94236672,"data_stored":94236672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509455,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42380,"kb_used_data":15360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329460,"statfs":{"total":96636764160,"available":96593367040,"internally_reserved":0,"allocated":15728640,"data_stored":15562249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542160,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45904,"kb_used_data":18884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94325936,"statfs":{"total":96636764160,"available":96589758464,"internally_reserved":0,"allocated":19337216,"data_stored":19170825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509456,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38644,"kb_used_data":11620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333196,"statfs":{"total":96636764160,"available":96597192704,"internally_reserved":0,"allocated":11898880,"data_stored":11732489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509456,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33596,"kb_used_data":6576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338244,"statfs":{"total":96636764160,"available":96602361856,"internally_reserved":0,"allocated":6733824,"data_stored":6567433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509456,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32444,"kb_used_data":5420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":5550080,"data_stored":5383689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509455,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40772,"kb_used_data":13752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331068,"statfs":{"total":96636764160,"available":96595013632,"internally_reserved":0,"allocated":14082048,"data_stored":13915657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542160,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36700,"kb_used_data":9680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335140,"statfs":{"total":96636764160,"available":96599183360,"internally_reserved":0,"allocated":9912320,"data_stored":9745929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574865,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39772,"kb_used_data":12752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332068,"statfs":{"total":96636764160,"available":96596037632,"internally_reserved":0,"allocated":13058048,"data_stored":12891657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11640832,"data_stored":11640832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19079168,"data_stored":19079168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9654272,"data_stored":9654272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:56:41.724 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T12:56:41.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:56:41.835+0000 7f6f4beaa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:56:41.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:56:41.847+0000 7f6f4beaa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:56:42.026 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:56:42.026 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T12:55:50.966805+0000","last_up_change":"2024-03-30T12:55:33.836470+0000","last_in_change":"2024-03-30T12:55:25.484082+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3146012944},{"type":"v1","addr":"172.21.15.67:6818","nonce":3146012944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3146012944},{"type":"v1","addr":"172.21.15.67:6821","nonce":3146012944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3146012944},{"type":"v1","addr":"172.21.15.67:6828","nonce":3146012944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":3146012944},{"type":"v1","addr":"172.21.15.67:6825","nonce":3146012944}]},"public_addr":"172.21.15.67:6818/3146012944","cluster_addr":"172.21.15.67:6821/3146012944","heartbeat_back_addr":"172.21.15.67:6828/3146012944","heartbeat_front_addr":"172.21.15.67:6825/3146012944","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6801","nonce":1504348724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6803","nonce":1504348724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6807","nonce":1504348724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6805","nonce":1504348724}]},"public_addr":"172.21.15.67:6801/1504348724","cluster_addr":"172.21.15.67:6803/1504348724","heartbeat_back_addr":"172.21.15.67:6807/1504348724","heartbeat_front_addr":"172.21.15.67:6805/1504348724","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1760616118},{"type":"v1","addr":"172.21.15.82:6803","nonce":1760616118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":1760616118},{"type":"v1","addr":"172.21.15.82:6807","nonce":1760616118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":1760616118},{"type":"v1","addr":"172.21.15.82:6815","nonce":1760616118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":1760616118},{"type":"v1","addr":"172.21.15.82:6811","nonce":1760616118}]},"public_addr":"172.21.15.82:6803/1760616118","cluster_addr":"172.21.15.82:6807/1760616118","heartbeat_back_addr":"172.21.15.82:6815/1760616118","heartbeat_front_addr":"172.21.15.82:6811/1760616118","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6817","nonce":3467856022}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6819","nonce":3467856022}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6823","nonce":3467856022}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6821","nonce":3467856022}]},"public_addr":"172.21.15.82:6817/3467856022","cluster_addr":"172.21.15.82:6819/3467856022","heartbeat_back_addr":"172.21.15.82:6823/3467856022","heartbeat_front_addr":"172.21.15.82:6821/3467856022","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6801","nonce":968508548}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6806","nonce":968508548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6814","nonce":968508548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6810","nonce":968508548}]},"public_addr":"172.21.15.82:6801/968508548","cluster_addr":"172.21.15.82:6806/968508548","heartbeat_back_addr":"172.21.15.82:6814/968508548","heartbeat_front_addr":"172.21.15.82:6810/968508548","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2471694698},{"type":"v1","addr":"172.21.15.82:6825","nonce":2471694698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2471694698},{"type":"v1","addr":"172.21.15.82:6827","nonce":2471694698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2471694698},{"type":"v1","addr":"172.21.15.82:6831","nonce":2471694698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2471694698},{"type":"v1","addr":"172.21.15.82:6829","nonce":2471694698}]},"public_addr":"172.21.15.82:6825/2471694698","cluster_addr":"172.21.15.82:6827/2471694698","heartbeat_back_addr":"172.21.15.82:6831/2471694698","heartbeat_front_addr":"172.21.15.82:6829/2471694698","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:55:30.919494+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:55:31.132465+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:55:30.980874+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:55:30.724117+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:55:30.946527+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:55:31.041350+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:55:31.124729+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:55:31.018547+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T12:56:42.046 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T12:56:42.046 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T12:56:42.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:56:42.163+0000 7f7d92885640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:56:42.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:56:42.183+0000 7f7d92885640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:56:42.356 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T12:56:42.378 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T12:56:42.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:56:42.487+0000 7fe4a870c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:56:42.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:56:42.503+0000 7fe4a870c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:56:42.683 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:56:42.683 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T12:56:42.701 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T12:56:42.701 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T12:56:42.701 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T12:56:42.702 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T12:56:42.702 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T12:56:42.702 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T12:56:42.702 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:56:42.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:56:42.803+0000 7f89620e6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:56:42.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:56:42.815+0000 7f89620e6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:56:42.985 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:56:42.985 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T12:56:43.001 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":53,"stamp":"2024-03-30T12:56:41.800871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":103,"num_read_kb":4624,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":310212,"kb_used_data":94044,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754664508,"statfs":{"total":773094113280,"available":772776456192,"internally_reserved":0,"allocated":96301056,"data_stored":94969928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004355"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.193120+0000","last_change":"2024-03-30T12:55:43.931163+0000","last_active":"2024-03-30T12:55:52.193120+0000","last_peered":"2024-03-30T12:55:52.193120+0000","last_clean":"2024-03-30T12:55:52.193120+0000","last_became_active":"2024-03-30T12:55:43.930823+0000","last_became_peered":"2024-03-30T12:55:43.930823+0000","last_unstale":"2024-03-30T12:55:52.193120+0000","last_undegraded":"2024-03-30T12:55:52.193120+0000","last_fullsized":"2024-03-30T12:55:52.193120+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:54.174180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:50.973407+0000","last_change":"2024-03-30T12:55:43.930916+0000","last_active":"2024-03-30T12:55:50.973407+0000","last_peered":"2024-03-30T12:55:50.973407+0000","last_clean":"2024-03-30T12:55:50.973407+0000","last_became_active":"2024-03-30T12:55:43.930591+0000","last_became_peered":"2024-03-30T12:55:43.930591+0000","last_unstale":"2024-03-30T12:55:50.973407+0000","last_undegraded":"2024-03-30T12:55:50.973407+0000","last_fullsized":"2024-03-30T12:55:50.973407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:07.237338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":34,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.600194+0000","last_change":"2024-03-30T12:55:43.931416+0000","last_active":"2024-03-30T12:55:51.600194+0000","last_peered":"2024-03-30T12:55:51.600194+0000","last_clean":"2024-03-30T12:55:51.600194+0000","last_became_active":"2024-03-30T12:55:43.931053+0000","last_became_peered":"2024-03-30T12:55:43.931053+0000","last_unstale":"2024-03-30T12:55:51.600194+0000","last_undegraded":"2024-03-30T12:55:51.600194+0000","last_fullsized":"2024-03-30T12:55:51.600194+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:02.687981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.600376+0000","last_change":"2024-03-30T12:55:43.938795+0000","last_active":"2024-03-30T12:55:51.600376+0000","last_peered":"2024-03-30T12:55:51.600376+0000","last_clean":"2024-03-30T12:55:51.600376+0000","last_became_active":"2024-03-30T12:55:43.938430+0000","last_became_peered":"2024-03-30T12:55:43.938430+0000","last_unstale":"2024-03-30T12:55:51.600376+0000","last_undegraded":"2024-03-30T12:55:51.600376+0000","last_fullsized":"2024-03-30T12:55:51.600376+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:44.051242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.951550+0000","last_change":"2024-03-30T12:55:43.934224+0000","last_active":"2024-03-30T12:55:51.951550+0000","last_peered":"2024-03-30T12:55:51.951550+0000","last_clean":"2024-03-30T12:55:51.951550+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T12:55:51.951550+0000","last_undegraded":"2024-03-30T12:55:51.951550+0000","last_fullsized":"2024-03-30T12:55:51.951550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:03.609885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":42,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.192795+0000","last_change":"2024-03-30T12:55:43.927603+0000","last_active":"2024-03-30T12:55:52.192795+0000","last_peered":"2024-03-30T12:55:52.192795+0000","last_clean":"2024-03-30T12:55:52.192795+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T12:55:52.192795+0000","last_undegraded":"2024-03-30T12:55:52.192795+0000","last_fullsized":"2024-03-30T12:55:52.192795+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:47.393835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":46,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.600181+0000","last_change":"2024-03-30T12:55:43.934166+0000","last_active":"2024-03-30T12:55:51.600181+0000","last_peered":"2024-03-30T12:55:51.600181+0000","last_clean":"2024-03-30T12:55:51.600181+0000","last_became_active":"2024-03-30T12:55:43.933833+0000","last_became_peered":"2024-03-30T12:55:43.933833+0000","last_unstale":"2024-03-30T12:55:51.600181+0000","last_undegraded":"2024-03-30T12:55:51.600181+0000","last_fullsized":"2024-03-30T12:55:51.600181+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:53.541028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":35,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:50.972914+0000","last_change":"2024-03-30T12:55:43.931142+0000","last_active":"2024-03-30T12:55:50.972914+0000","last_peered":"2024-03-30T12:55:50.972914+0000","last_clean":"2024-03-30T12:55:50.972914+0000","last_became_active":"2024-03-30T12:55:43.930722+0000","last_became_peered":"2024-03-30T12:55:43.930722+0000","last_unstale":"2024-03-30T12:55:50.972914+0000","last_undegraded":"2024-03-30T12:55:50.972914+0000","last_fullsized":"2024-03-30T12:55:50.972914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:47.641687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.192310+0000","last_change":"2024-03-30T12:55:43.933369+0000","last_active":"2024-03-30T12:55:52.192310+0000","last_peered":"2024-03-30T12:55:52.192310+0000","last_clean":"2024-03-30T12:55:52.192310+0000","last_became_active":"2024-03-30T12:55:43.932863+0000","last_became_peered":"2024-03-30T12:55:43.932863+0000","last_unstale":"2024-03-30T12:55:52.192310+0000","last_undegraded":"2024-03-30T12:55:52.192310+0000","last_fullsized":"2024-03-30T12:55:52.192310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:59.005903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:50.973469+0000","last_change":"2024-03-30T12:55:43.936909+0000","last_active":"2024-03-30T12:55:50.973469+0000","last_peered":"2024-03-30T12:55:50.973469+0000","last_clean":"2024-03-30T12:55:50.973469+0000","last_became_active":"2024-03-30T12:55:43.936547+0000","last_became_peered":"2024-03-30T12:55:43.936547+0000","last_unstale":"2024-03-30T12:55:50.973469+0000","last_undegraded":"2024-03-30T12:55:50.973469+0000","last_fullsized":"2024-03-30T12:55:50.973469+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:07.670156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":46,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.193092+0000","last_change":"2024-03-30T12:55:43.936225+0000","last_active":"2024-03-30T12:55:52.193092+0000","last_peered":"2024-03-30T12:55:52.193092+0000","last_clean":"2024-03-30T12:55:52.193092+0000","last_became_active":"2024-03-30T12:55:43.935038+0000","last_became_peered":"2024-03-30T12:55:43.935038+0000","last_unstale":"2024-03-30T12:55:52.193092+0000","last_undegraded":"2024-03-30T12:55:52.193092+0000","last_fullsized":"2024-03-30T12:55:52.193092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:53.215113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":54,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.193093+0000","last_change":"2024-03-30T12:55:43.931046+0000","last_active":"2024-03-30T12:55:52.193093+0000","last_peered":"2024-03-30T12:55:52.193093+0000","last_clean":"2024-03-30T12:55:52.193093+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T12:55:52.193093+0000","last_undegraded":"2024-03-30T12:55:52.193093+0000","last_fullsized":"2024-03-30T12:55:52.193093+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:09.603399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":39,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.951564+0000","last_change":"2024-03-30T12:55:43.936654+0000","last_active":"2024-03-30T12:55:51.951564+0000","last_peered":"2024-03-30T12:55:51.951564+0000","last_clean":"2024-03-30T12:55:51.951564+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T12:55:51.951564+0000","last_undegraded":"2024-03-30T12:55:51.951564+0000","last_fullsized":"2024-03-30T12:55:51.951564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:53.756917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.600158+0000","last_change":"2024-03-30T12:55:43.935508+0000","last_active":"2024-03-30T12:55:51.600158+0000","last_peered":"2024-03-30T12:55:51.600158+0000","last_clean":"2024-03-30T12:55:51.600158+0000","last_became_active":"2024-03-30T12:55:43.934495+0000","last_became_peered":"2024-03-30T12:55:43.934495+0000","last_unstale":"2024-03-30T12:55:51.600158+0000","last_undegraded":"2024-03-30T12:55:51.600158+0000","last_fullsized":"2024-03-30T12:55:51.600158+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:48.367617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":39,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:51.951580+0000","last_change":"2024-03-30T12:55:43.935230+0000","last_active":"2024-03-30T12:55:51.951580+0000","last_peered":"2024-03-30T12:55:51.951580+0000","last_clean":"2024-03-30T12:55:51.951580+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T12:55:51.951580+0000","last_undegraded":"2024-03-30T12:55:51.951580+0000","last_fullsized":"2024-03-30T12:55:51.951580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:47.344018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:50.972853+0000","last_change":"2024-03-30T12:55:43.926406+0000","last_active":"2024-03-30T12:55:50.972853+0000","last_peered":"2024-03-30T12:55:50.972853+0000","last_clean":"2024-03-30T12:55:50.972853+0000","last_became_active":"2024-03-30T12:55:43.925945+0000","last_became_peered":"2024-03-30T12:55:43.925945+0000","last_unstale":"2024-03-30T12:55:50.972853+0000","last_undegraded":"2024-03-30T12:55:50.972853+0000","last_fullsized":"2024-03-30T12:55:50.972853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:08.300071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":103,"num_read_kb":4624,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94236672,"data_stored":94236672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509456,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42380,"kb_used_data":15360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329460,"statfs":{"total":96636764160,"available":96593367040,"internally_reserved":0,"allocated":15728640,"data_stored":15562249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542160,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45904,"kb_used_data":18884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94325936,"statfs":{"total":96636764160,"available":96589758464,"internally_reserved":0,"allocated":19337216,"data_stored":19170825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":15,"seq":64424509457,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38644,"kb_used_data":11620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333196,"statfs":{"total":96636764160,"available":96597192704,"internally_reserved":0,"allocated":11898880,"data_stored":11732489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509457,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33596,"kb_used_data":6576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338244,"statfs":{"total":96636764160,"available":96602361856,"internally_reserved":0,"allocated":6733824,"data_stored":6567433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509457,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32444,"kb_used_data":5420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":5550080,"data_stored":5383689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509456,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40772,"kb_used_data":13752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331068,"statfs":{"total":96636764160,"available":96595013632,"internally_reserved":0,"allocated":14082048,"data_stored":13915657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542160,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36700,"kb_used_data":9680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335140,"statfs":{"total":96636764160,"available":96599183360,"internally_reserved":0,"allocated":9912320,"data_stored":9745929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574865,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39772,"kb_used_data":12752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332068,"statfs":{"total":96636764160,"available":96596037632,"internally_reserved":0,"allocated":13058048,"data_stored":12891657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11640832,"data_stored":11640832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19079168,"data_stored":19079168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9654272,"data_stored":9654272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:56:43.002 INFO:tasks.thrashosds.thrasher:Choosing PG 1.5 with acting set [4, 2, 6] 2024-03-30T12:56:43.002 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T12:56:43.002 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-30T12:56:43.002 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T12:56:43.083 INFO:tasks.ceph.osd.4:Stopped 2024-03-30T12:56:43.083 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-30T12:56:43.083 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 4 2024-03-30T12:56:43.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:56:43.203+0000 7f80872b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:56:43.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:56:43.215+0000 7f80872b1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:56:43.400 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T12:56:43.395+0000 7fbb38f08640 -1 mon.a@0(leader).osd e26 definitely_dead 0 2024-03-30T12:56:44.112 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.4. 2024-03-30T12:56:44.139 INFO:tasks.thrashosds.thrasher:dead_osds=[4], live_osds=[0, 1, 2, 3, 5, 6, 7] 2024-03-30T12:56:44.139 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T12:56:44.140 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:56:44.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:56:44.251+0000 7fbf356a5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:56:44.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:56:44.263+0000 7fbf356a5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:56:44.443 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:56:44.444 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T12:56:44.463 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":56,"stamp":"2024-03-30T12:56:44.109627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":102,"num_read_kb":4623,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":271580,"kb_used_data":82436,"kb_used_omap":10,"kb_used_meta":189045,"kb_avail":660331300,"statfs":{"total":676457349120,"available":676179251200,"internally_reserved":0,"allocated":84414464,"data_stored":83243070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11161,"internal_metadata":193582183},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.310862"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.193120+0000","last_change":"2024-03-30T12:55:43.931163+0000","last_active":"2024-03-30T12:55:52.193120+0000","last_peered":"2024-03-30T12:55:52.193120+0000","last_clean":"2024-03-30T12:55:52.193120+0000","last_became_active":"2024-03-30T12:55:43.930823+0000","last_became_peered":"2024-03-30T12:55:43.930823+0000","last_unstale":"2024-03-30T12:55:52.193120+0000","last_undegraded":"2024-03-30T12:55:52.193120+0000","last_fullsized":"2024-03-30T12:55:52.193120+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:54.174180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":21,"reported_epoch":26,"state":"peering","last_fresh":"2024-03-30T12:56:43.104372+0000","last_change":"2024-03-30T12:56:43.104372+0000","last_active":"2024-03-30T12:56:43.104372+0000","last_peered":"2024-03-30T12:55:47.511068+0000","last_clean":"2024-03-30T12:55:47.511068+0000","last_became_active":"2024-03-30T12:55:43.930591+0000","last_became_peered":"2024-03-30T12:55:43.930591+0000","last_unstale":"2024-03-30T12:56:43.104372+0000","last_undegraded":"2024-03-30T12:56:43.104372+0000","last_fullsized":"2024-03-30T12:56:43.104372+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,7,3],"acting":[2147483647,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":35,"reported_epoch":25,"state":"peering","last_fresh":"2024-03-30T12:55:51.600267+0000","last_change":"2024-03-30T12:56:43.105646+0000","last_active":"2024-03-30T12:56:43.105646+0000","last_peered":"2024-03-30T12:55:51.600267+0000","last_clean":"2024-03-30T12:55:51.600267+0000","last_became_active":"2024-03-30T12:55:43.931053+0000","last_became_peered":"2024-03-30T12:55:43.931053+0000","last_unstale":"2024-03-30T12:55:51.600267+0000","last_undegraded":"2024-03-30T12:55:51.600267+0000","last_fullsized":"2024-03-30T12:55:51.600267+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2147483647,1],"acting":[7,2147483647,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-30T12:56:43.103876+0000","last_change":"2024-03-30T12:55:43.938795+0000","last_active":"2024-03-30T12:56:43.103876+0000","last_peered":"2024-03-30T12:56:43.103876+0000","last_clean":"2024-03-30T12:56:43.103876+0000","last_became_active":"2024-03-30T12:55:43.938430+0000","last_became_peered":"2024-03-30T12:55:43.938430+0000","last_unstale":"2024-03-30T12:56:43.103876+0000","last_undegraded":"2024-03-30T12:56:43.103876+0000","last_fullsized":"2024-03-30T12:56:43.103876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:44.051242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-30T12:56:43.117621+0000","last_change":"2024-03-30T12:55:43.934224+0000","last_active":"2024-03-30T12:56:43.117621+0000","last_peered":"2024-03-30T12:56:43.117621+0000","last_clean":"2024-03-30T12:56:43.117621+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T12:56:43.117621+0000","last_undegraded":"2024-03-30T12:56:43.117621+0000","last_fullsized":"2024-03-30T12:56:43.117621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:03.609885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":42,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.192795+0000","last_change":"2024-03-30T12:55:43.927603+0000","last_active":"2024-03-30T12:55:52.192795+0000","last_peered":"2024-03-30T12:55:52.192795+0000","last_clean":"2024-03-30T12:55:52.192795+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T12:55:52.192795+0000","last_undegraded":"2024-03-30T12:55:52.192795+0000","last_fullsized":"2024-03-30T12:55:52.192795+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:47.393835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":47,"reported_epoch":25,"state":"peering","last_fresh":"2024-03-30T12:55:51.600259+0000","last_change":"2024-03-30T12:56:43.105328+0000","last_active":"2024-03-30T12:56:43.105328+0000","last_peered":"2024-03-30T12:55:51.600259+0000","last_clean":"2024-03-30T12:55:51.600259+0000","last_became_active":"2024-03-30T12:55:43.933833+0000","last_became_peered":"2024-03-30T12:55:43.933833+0000","last_unstale":"2024-03-30T12:55:51.600259+0000","last_undegraded":"2024-03-30T12:55:51.600259+0000","last_fullsized":"2024-03-30T12:55:51.600259+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2147483647],"acting":[7,0,2147483647],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":35,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:50.972914+0000","last_change":"2024-03-30T12:55:43.931142+0000","last_active":"2024-03-30T12:55:50.972914+0000","last_peered":"2024-03-30T12:55:50.972914+0000","last_clean":"2024-03-30T12:55:50.972914+0000","last_became_active":"2024-03-30T12:55:43.930722+0000","last_became_peered":"2024-03-30T12:55:43.930722+0000","last_unstale":"2024-03-30T12:55:50.972914+0000","last_undegraded":"2024-03-30T12:55:50.972914+0000","last_fullsized":"2024-03-30T12:55:50.972914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:47.641687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.192310+0000","last_change":"2024-03-30T12:55:43.933369+0000","last_active":"2024-03-30T12:55:52.192310+0000","last_peered":"2024-03-30T12:55:52.192310+0000","last_clean":"2024-03-30T12:55:52.192310+0000","last_became_active":"2024-03-30T12:55:43.932863+0000","last_became_peered":"2024-03-30T12:55:43.932863+0000","last_unstale":"2024-03-30T12:55:52.192310+0000","last_undegraded":"2024-03-30T12:55:52.192310+0000","last_fullsized":"2024-03-30T12:55:52.192310+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:59.005903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":26,"reported_epoch":25,"state":"stale+active+clean","last_fresh":"2024-03-30T12:55:50.973469+0000","last_change":"2024-03-30T12:55:43.936909+0000","last_active":"2024-03-30T12:55:50.973469+0000","last_peered":"2024-03-30T12:55:50.973469+0000","last_clean":"2024-03-30T12:55:50.973469+0000","last_became_active":"2024-03-30T12:55:43.936547+0000","last_became_peered":"2024-03-30T12:55:43.936547+0000","last_unstale":"2024-03-30T12:56:43.101924+0000","last_undegraded":"2024-03-30T12:55:50.973469+0000","last_fullsized":"2024-03-30T12:55:50.973469+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:07.670156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":46,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.193092+0000","last_change":"2024-03-30T12:55:43.936225+0000","last_active":"2024-03-30T12:55:52.193092+0000","last_peered":"2024-03-30T12:55:52.193092+0000","last_clean":"2024-03-30T12:55:52.193092+0000","last_became_active":"2024-03-30T12:55:43.935038+0000","last_became_peered":"2024-03-30T12:55:43.935038+0000","last_unstale":"2024-03-30T12:55:52.193092+0000","last_undegraded":"2024-03-30T12:55:52.193092+0000","last_fullsized":"2024-03-30T12:55:52.193092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:53.215113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":54,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:52.193093+0000","last_change":"2024-03-30T12:55:43.931046+0000","last_active":"2024-03-30T12:55:52.193093+0000","last_peered":"2024-03-30T12:55:52.193093+0000","last_clean":"2024-03-30T12:55:52.193093+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T12:55:52.193093+0000","last_undegraded":"2024-03-30T12:55:52.193093+0000","last_fullsized":"2024-03-30T12:55:52.193093+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:09.603399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":41,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-30T12:56:43.117660+0000","last_change":"2024-03-30T12:55:43.936654+0000","last_active":"2024-03-30T12:56:43.117660+0000","last_peered":"2024-03-30T12:56:43.117660+0000","last_clean":"2024-03-30T12:56:43.117660+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T12:56:43.117660+0000","last_undegraded":"2024-03-30T12:56:43.117660+0000","last_fullsized":"2024-03-30T12:56:43.117660+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:53.756917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-30T12:56:43.103817+0000","last_change":"2024-03-30T12:55:43.935508+0000","last_active":"2024-03-30T12:56:43.103817+0000","last_peered":"2024-03-30T12:56:43.103817+0000","last_clean":"2024-03-30T12:56:43.103817+0000","last_became_active":"2024-03-30T12:55:43.934495+0000","last_became_peered":"2024-03-30T12:55:43.934495+0000","last_unstale":"2024-03-30T12:56:43.103817+0000","last_undegraded":"2024-03-30T12:56:43.103817+0000","last_fullsized":"2024-03-30T12:56:43.103817+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:48.367617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":41,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-30T12:56:43.117681+0000","last_change":"2024-03-30T12:55:43.935230+0000","last_active":"2024-03-30T12:56:43.117681+0000","last_peered":"2024-03-30T12:56:43.117681+0000","last_clean":"2024-03-30T12:56:43.117681+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T12:56:43.117681+0000","last_undegraded":"2024-03-30T12:56:43.117681+0000","last_fullsized":"2024-03-30T12:56:43.117681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:56:47.344018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-30T12:55:50.972853+0000","last_change":"2024-03-30T12:55:43.926406+0000","last_active":"2024-03-30T12:55:50.972853+0000","last_peered":"2024-03-30T12:55:50.972853+0000","last_clean":"2024-03-30T12:55:50.972853+0000","last_became_active":"2024-03-30T12:55:43.925945+0000","last_became_peered":"2024-03-30T12:55:43.925945+0000","last_unstale":"2024-03-30T12:55:50.972853+0000","last_undegraded":"2024-03-30T12:55:50.972853+0000","last_fullsized":"2024-03-30T12:55:50.972853+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:08.300071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":102,"num_read_kb":4623,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94236672,"data_stored":94236672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509456,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42380,"kb_used_data":15360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329460,"statfs":{"total":96636764160,"available":96593367040,"internally_reserved":0,"allocated":15728640,"data_stored":15562249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542161,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45916,"kb_used_data":18896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94325924,"statfs":{"total":96636764160,"available":96589746176,"internally_reserved":0,"allocated":19349504,"data_stored":19176456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509457,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33596,"kb_used_data":6576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338244,"statfs":{"total":96636764160,"available":96602361856,"internally_reserved":0,"allocated":6733824,"data_stored":6567433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509457,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32444,"kb_used_data":5420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339396,"statfs":{"total":96636764160,"available":96603541504,"internally_reserved":0,"allocated":5550080,"data_stored":5383689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509456,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40772,"kb_used_data":13752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331068,"statfs":{"total":96636764160,"available":96595013632,"internally_reserved":0,"allocated":14082048,"data_stored":13915657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542161,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36700,"kb_used_data":9680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335140,"statfs":{"total":96636764160,"available":96599183360,"internally_reserved":0,"allocated":9912320,"data_stored":9745929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39772,"kb_used_data":12752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332068,"statfs":{"total":96636764160,"available":96596037632,"internally_reserved":0,"allocated":13058048,"data_stored":12891657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11640832,"data_stored":11640832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19079168,"data_stored":19079168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9654272,"data_stored":9654272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:56:44.464 INFO:tasks.ceph.ceph_manager.ceph:PG 1.e is not active or peered 2024-03-30T12:56:44.464 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.e', 'version': "22'3", 'reported_seq': 21, 'reported_epoch': 26, 'state': 'peering', 'last_fresh': '2024-03-30T12:56:43.104372+0000', 'last_change': '2024-03-30T12:56:43.104372+0000', 'last_active': '2024-03-30T12:56:43.104372+0000', 'last_peered': '2024-03-30T12:55:47.511068+0000', 'last_clean': '2024-03-30T12:55:47.511068+0000', 'last_became_active': '2024-03-30T12:55:43.930591+0000', 'last_became_peered': '2024-03-30T12:55:43.930591+0000', 'last_unstale': '2024-03-30T12:56:43.104372+0000', 'last_undegraded': '2024-03-30T12:56:43.104372+0000', 'last_fullsized': '2024-03-30T12:56:43.104372+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 20, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-30T12:55:42.900074+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-30T12:55:42.900074+0000', 'last_clean_scrub_stamp': '2024-03-30T12:55:42.900074+0000', 'objects_scrubbed': 0, 'log_size': 3, 'log_dups_size': 0, 'ondisk_log_size': 3, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2655473, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 3, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 0, 'num_write': 4, 'num_write_kb': 1244, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 7, 3], 'acting': [2147483647, 7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-30T12:56:44.465 INFO:tasks.ceph.ceph_manager.ceph:PG 1.d is not active or peered 2024-03-30T12:56:44.465 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.d', 'version': "22'9", 'reported_seq': 35, 'reported_epoch': 25, 'state': 'peering', 'last_fresh': '2024-03-30T12:55:51.600267+0000', 'last_change': '2024-03-30T12:56:43.105646+0000', 'last_active': '2024-03-30T12:56:43.105646+0000', 'last_peered': '2024-03-30T12:55:51.600267+0000', 'last_clean': '2024-03-30T12:55:51.600267+0000', 'last_became_active': '2024-03-30T12:55:43.931053+0000', 'last_became_peered': '2024-03-30T12:55:43.931053+0000', 'last_unstale': '2024-03-30T12:55:51.600267+0000', 'last_undegraded': '2024-03-30T12:55:51.600267+0000', 'last_fullsized': '2024-03-30T12:55:51.600267+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 20, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-30T12:55:42.900074+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-30T12:55:42.900074+0000', 'last_clean_scrub_stamp': '2024-03-30T12:55:42.900074+0000', 'objects_scrubbed': 0, 'log_size': 9, 'log_dups_size': 0, 'ondisk_log_size': 9, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7815156, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 9, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 6, 'num_read_kb': 3, 'num_write': 12, 'num_write_kb': 3459, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2147483647, 1], 'acting': [7, 2147483647, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-30T12:56:44.465 INFO:tasks.ceph.ceph_manager.ceph:PG 1.2 is not active or peered 2024-03-30T12:56:44.465 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.2', 'version': "22'18", 'reported_seq': 47, 'reported_epoch': 25, 'state': 'peering', 'last_fresh': '2024-03-30T12:55:51.600259+0000', 'last_change': '2024-03-30T12:56:43.105328+0000', 'last_active': '2024-03-30T12:56:43.105328+0000', 'last_peered': '2024-03-30T12:55:51.600259+0000', 'last_clean': '2024-03-30T12:55:51.600259+0000', 'last_became_active': '2024-03-30T12:55:43.933833+0000', 'last_became_peered': '2024-03-30T12:55:43.933833+0000', 'last_unstale': '2024-03-30T12:55:51.600259+0000', 'last_undegraded': '2024-03-30T12:55:51.600259+0000', 'last_fullsized': '2024-03-30T12:55:51.600259+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 20, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-30T12:55:42.900074+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-30T12:55:42.900074+0000', 'last_clean_scrub_stamp': '2024-03-30T12:55:42.900074+0000', 'objects_scrubbed': 0, 'log_size': 18, 'log_dups_size': 0, 'ondisk_log_size': 18, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14638699, 'num_objects': 6, 'num_object_clones': 0, 'num_object_copies': 18, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6, 'num_whiteouts': 0, 'num_read': 12, 'num_read_kb': 6, 'num_write': 24, 'num_write_kb': 6991, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0, 2147483647], 'acting': [7, 0, 2147483647], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-30T12:56:44.465 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T12:57:09.466 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T12:57:09.467 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:57:09.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:09.594+0000 7f4ca5961640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:09.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:09.606+0000 7f4ca5961640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:09.782 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:57:09.783 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T12:57:09.800 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2024-03-30T12:57:07.808342+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":101,"num_read_kb":4622,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44773254,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":284532,"kb_used_data":95392,"kb_used_omap":10,"kb_used_meta":189045,"kb_avail":660318348,"statfs":{"total":676457349120,"available":676165988352,"internally_reserved":0,"allocated":97681408,"data_stored":96090224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11161,"internal_metadata":193582183},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002797"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":60,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-30T12:57:01.404967+0000","last_change":"2024-03-30T12:56:45.180421+0000","last_active":"2024-03-30T12:57:01.404967+0000","last_peered":"2024-03-30T12:57:01.404967+0000","last_clean":"2024-03-30T12:57:01.404967+0000","last_became_active":"2024-03-30T12:56:45.129782+0000","last_became_peered":"2024-03-30T12:56:45.129782+0000","last_unstale":"2024-03-30T12:57:01.404967+0000","last_undegraded":"2024-03-30T12:57:01.404967+0000","last_fullsized":"2024-03-30T12:57:01.404967+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:10.369081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":35,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-30T12:57:01.255095+0000","last_change":"2024-03-30T12:56:54.950132+0000","last_active":"2024-03-30T12:57:01.255095+0000","last_peered":"2024-03-30T12:57:01.255095+0000","last_clean":"2024-03-30T12:57:01.255095+0000","last_became_active":"2024-03-30T12:56:45.124200+0000","last_became_peered":"2024-03-30T12:56:45.124200+0000","last_unstale":"2024-03-30T12:57:01.255095+0000","last_undegraded":"2024-03-30T12:57:01.255095+0000","last_fullsized":"2024-03-30T12:57:01.255095+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:56:54.949876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:54.949876+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:15.852450+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":73,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-30T12:57:01.250469+0000","last_change":"2024-03-30T12:56:59.916726+0000","last_active":"2024-03-30T12:57:01.250469+0000","last_peered":"2024-03-30T12:57:01.250469+0000","last_clean":"2024-03-30T12:57:01.250469+0000","last_became_active":"2024-03-30T12:56:45.127817+0000","last_became_peered":"2024-03-30T12:56:45.127817+0000","last_unstale":"2024-03-30T12:57:01.250469+0000","last_undegraded":"2024-03-30T12:57:01.250469+0000","last_fullsized":"2024-03-30T12:57:01.250469+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:56:59.916447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:59.916447+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:24.384919+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-30T12:57:01.250361+0000","last_change":"2024-03-30T12:56:44.900397+0000","last_active":"2024-03-30T12:57:01.250361+0000","last_peered":"2024-03-30T12:57:01.250361+0000","last_clean":"2024-03-30T12:57:01.250361+0000","last_became_active":"2024-03-30T12:55:43.938430+0000","last_became_peered":"2024-03-30T12:55:43.938430+0000","last_unstale":"2024-03-30T12:57:01.250361+0000","last_undegraded":"2024-03-30T12:57:01.250361+0000","last_fullsized":"2024-03-30T12:57:01.250361+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:56:44.900077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:44.900077+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:04.497239+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":45,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-30T12:57:01.405716+0000","last_change":"2024-03-30T12:55:43.934224+0000","last_active":"2024-03-30T12:57:01.405716+0000","last_peered":"2024-03-30T12:57:01.405716+0000","last_clean":"2024-03-30T12:57:01.405716+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T12:57:01.405716+0000","last_undegraded":"2024-03-30T12:57:01.405716+0000","last_fullsized":"2024-03-30T12:57:01.405716+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:03.609885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":68,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-30T12:57:01.405350+0000","last_change":"2024-03-30T12:56:48.287112+0000","last_active":"2024-03-30T12:57:01.405350+0000","last_peered":"2024-03-30T12:57:01.405350+0000","last_clean":"2024-03-30T12:57:01.405350+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T12:57:01.405350+0000","last_undegraded":"2024-03-30T12:57:01.405350+0000","last_fullsized":"2024-03-30T12:57:01.405350+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:56:48.286816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:48.286816+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:59.500682+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":91,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-30T12:57:02.954552+0000","last_change":"2024-03-30T12:57:02.954696+0000","last_active":"2024-03-30T12:57:02.954552+0000","last_peered":"2024-03-30T12:57:02.954552+0000","last_clean":"2024-03-30T12:57:02.954552+0000","last_became_active":"2024-03-30T12:56:45.136283+0000","last_became_peered":"2024-03-30T12:56:45.136283+0000","last_unstale":"2024-03-30T12:57:02.954552+0000","last_undegraded":"2024-03-30T12:57:02.954552+0000","last_fullsized":"2024-03-30T12:57:02.954552+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T12:57:02.954261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:02.954261+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:06.552924+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":61,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-30T12:57:01.253446+0000","last_change":"2024-03-30T12:56:58.446137+0000","last_active":"2024-03-30T12:57:01.253446+0000","last_peered":"2024-03-30T12:57:01.253446+0000","last_clean":"2024-03-30T12:57:01.253446+0000","last_became_active":"2024-03-30T12:55:43.930722+0000","last_became_peered":"2024-03-30T12:55:43.930722+0000","last_unstale":"2024-03-30T12:57:01.253446+0000","last_undegraded":"2024-03-30T12:57:01.253446+0000","last_fullsized":"2024-03-30T12:57:01.253446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:56:58.445737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:58.445737+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:21.311535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-30T12:57:01.405361+0000","last_change":"2024-03-30T12:57:00.268436+0000","last_active":"2024-03-30T12:57:01.405361+0000","last_peered":"2024-03-30T12:57:01.405361+0000","last_clean":"2024-03-30T12:57:01.405361+0000","last_became_active":"2024-03-30T12:55:43.932863+0000","last_became_peered":"2024-03-30T12:55:43.932863+0000","last_unstale":"2024-03-30T12:57:01.405361+0000","last_undegraded":"2024-03-30T12:57:01.405361+0000","last_fullsized":"2024-03-30T12:57:01.405361+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:00.268171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:00.268171+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:11.652557+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":27,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-30T12:57:01.405671+0000","last_change":"2024-03-30T12:56:45.167067+0000","last_active":"2024-03-30T12:57:01.405671+0000","last_peered":"2024-03-30T12:57:01.405671+0000","last_clean":"2024-03-30T12:57:01.405671+0000","last_became_active":"2024-03-30T12:56:45.130306+0000","last_became_peered":"2024-03-30T12:56:45.130306+0000","last_unstale":"2024-03-30T12:57:01.405671+0000","last_undegraded":"2024-03-30T12:57:01.405671+0000","last_fullsized":"2024-03-30T12:57:01.405671+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:08.449904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":89,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-30T12:57:01.404956+0000","last_change":"2024-03-30T12:57:00.040236+0000","last_active":"2024-03-30T12:57:01.404956+0000","last_peered":"2024-03-30T12:57:01.404956+0000","last_clean":"2024-03-30T12:57:01.404956+0000","last_became_active":"2024-03-30T12:56:45.128939+0000","last_became_peered":"2024-03-30T12:56:45.128939+0000","last_unstale":"2024-03-30T12:57:01.404956+0000","last_undegraded":"2024-03-30T12:57:01.404956+0000","last_fullsized":"2024-03-30T12:57:01.404956+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:57:00.039850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:00.039850+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:14.822472+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":72,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-30T12:57:01.405091+0000","last_change":"2024-03-30T12:55:43.931046+0000","last_active":"2024-03-30T12:57:01.405091+0000","last_peered":"2024-03-30T12:57:01.405091+0000","last_clean":"2024-03-30T12:57:01.405091+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T12:57:01.405091+0000","last_undegraded":"2024-03-30T12:57:01.405091+0000","last_fullsized":"2024-03-30T12:57:01.405091+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:09.603399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":65,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-30T12:57:01.405716+0000","last_change":"2024-03-30T12:56:56.068730+0000","last_active":"2024-03-30T12:57:01.405716+0000","last_peered":"2024-03-30T12:57:01.405716+0000","last_clean":"2024-03-30T12:57:01.405716+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T12:57:01.405716+0000","last_undegraded":"2024-03-30T12:57:01.405716+0000","last_fullsized":"2024-03-30T12:57:01.405716+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:56:56.068479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:56.068479+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:08.832176+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":52,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-30T12:57:01.250394+0000","last_change":"2024-03-30T12:56:56.870958+0000","last_active":"2024-03-30T12:57:01.250394+0000","last_peered":"2024-03-30T12:57:01.250394+0000","last_clean":"2024-03-30T12:57:01.250394+0000","last_became_active":"2024-03-30T12:55:43.934495+0000","last_became_peered":"2024-03-30T12:55:43.934495+0000","last_unstale":"2024-03-30T12:57:01.250394+0000","last_undegraded":"2024-03-30T12:57:01.250394+0000","last_fullsized":"2024-03-30T12:57:01.250394+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:56:56.870694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:56.870694+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:20.371042+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":65,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-30T12:57:01.405739+0000","last_change":"2024-03-30T12:56:50.034844+0000","last_active":"2024-03-30T12:57:01.405739+0000","last_peered":"2024-03-30T12:57:01.405739+0000","last_clean":"2024-03-30T12:57:01.405739+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T12:57:01.405739+0000","last_undegraded":"2024-03-30T12:57:01.405739+0000","last_fullsized":"2024-03-30T12:57:01.405739+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:56:50.034645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:50.034645+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:07.024201+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-03-30T12:57:01.253121+0000","last_change":"2024-03-30T12:55:43.926406+0000","last_active":"2024-03-30T12:57:01.253121+0000","last_peered":"2024-03-30T12:57:01.253121+0000","last_clean":"2024-03-30T12:57:01.253121+0000","last_became_active":"2024-03-30T12:55:43.925945+0000","last_became_peered":"2024-03-30T12:55:43.925945+0000","last_unstale":"2024-03-30T12:57:01.253121+0000","last_undegraded":"2024-03-30T12:57:01.253121+0000","last_fullsized":"2024-03-30T12:57:01.253121+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:55:42.900074+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:08.300071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":101,"num_read_kb":4622,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44773254,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106741760,"data_stored":106741760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509461,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42488,"kb_used_data":15468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329352,"statfs":{"total":96636764160,"available":96593256448,"internally_reserved":0,"allocated":15839232,"data_stored":15611920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542165,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46012,"kb_used_data":18992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94325828,"statfs":{"total":96636764160,"available":96589647872,"internally_reserved":0,"allocated":19447808,"data_stored":19220496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509462,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40460,"kb_used_data":13440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331380,"statfs":{"total":96636764160,"available":96595333120,"internally_reserved":0,"allocated":13762560,"data_stored":13535248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509462,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33640,"kb_used_data":6620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338200,"statfs":{"total":96636764160,"available":96602316800,"internally_reserved":0,"allocated":6778880,"data_stored":6551568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509461,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41384,"kb_used_data":14364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330456,"statfs":{"total":96636764160,"available":96594386944,"internally_reserved":0,"allocated":14708736,"data_stored":14481424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542165,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36808,"kb_used_data":9788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335032,"statfs":{"total":96636764160,"available":96599072768,"internally_reserved":0,"allocated":10022912,"data_stored":9795600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43740,"kb_used_data":16720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94328100,"statfs":{"total":96636764160,"available":96591974400,"internally_reserved":0,"allocated":17121280,"data_stored":16893968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14340096,"data_stored":14340096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6410240,"data_stored":6410240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13393920,"data_stored":13393920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16752640,"data_stored":16752640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11640832,"data_stored":11640832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19079168,"data_stored":19079168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9654272,"data_stored":9654272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:57:09.802 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T12:57:09.802 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T12:57:09.802 INFO:tasks.thrashosds.thrasher:Reviving osd 4 2024-03-30T12:57:09.802 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-30T12:57:09.802 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-30T12:57:09.805 INFO:tasks.ceph.osd.4:Started 2024-03-30T12:57:09.805 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-30T12:57:09.868 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T12:57:09.864+0000 7fb1770647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:09.877 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T12:57:09.872+0000 7fb1770647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:09.877 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T12:57:09.872+0000 7fb1770647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:09.926 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T12:57:09.926 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T12:57:09.926 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T12:57:09.926 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T12:57:09.926 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T12:57:09.926 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T12:57:09.926 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-03-30T12:57:09.926 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-03-30T12:57:09.926 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-03-30T12:57:09.926 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-03-30T12:57:09.927 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-03-30T12:57:09.927 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T12:57:09.928 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T12:57:09.929 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-03-30T12:57:10.423 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T12:57:10.420+0000 7fb1770647c0 -1 Falling back to public interface 2024-03-30T12:57:14.158 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T12:57:14.156+0000 7fb1770647c0 -1 osd.4 25 log_to_monitors true 2024-03-30T12:57:14.930 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-30T12:57:14.931 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T12:57:14.928+0000 7fb16ee39640 -1 osd.4 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T12:57:15.060 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T12:57:15.061 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T12:57:15.061 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T12:57:15.061 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T12:57:15.074 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T12:57:15.074 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-03-30T12:57:15.075 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 4 2024-03-30T12:57:15.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:15.202+0000 7f1df04a1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:15.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:15.214+0000 7f1df04a1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:15.398 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T12:57:15.398+0000 7fbb38f08640 -1 mon.a@0(leader).osd e35 definitely_dead 0 2024-03-30T12:57:15.898 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.4. 2024-03-30T12:57:15.928 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-03-30T12:57:15.928 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T12:57:16.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:16.046+0000 7f726d1e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:16.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:16.062+0000 7f726d1e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:16.920 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T12:57:16.950 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T12:57:17.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:17.066+0000 7f734e83d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:17.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:17.082+0000 7f734e83d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:17.926 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T12:57:17.953 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T12:57:18.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:18.074+0000 7f0c0e66b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:18.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:18.094+0000 7f0c0e66b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:18.930 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T12:57:18.959 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T12:57:19.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:19.078+0000 7f07d226c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:19.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:19.094+0000 7f07d226c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:19.954 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T12:57:19.980 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T12:57:20.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:20.102+0000 7f4e160ba640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:20.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:20.118+0000 7f4e160ba640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:20.956 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T12:57:20.983 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T12:57:21.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:21.102+0000 7fede1a09640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:21.118+0000 7fede1a09640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:21.970 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T12:57:21.996 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T12:57:22.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:22.113+0000 7fe50da49640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:22.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:22.129+0000 7fe50da49640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:22.974 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T12:57:23.001 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T12:57:23.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:23.121+0000 7f3bebdc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:23.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:23.141+0000 7f3bebdc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:24.004 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T12:57:24.027 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T12:57:24.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:24.137+0000 7f7f8319e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:24.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:24.153+0000 7f7f8319e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:25.005 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T12:57:25.032 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T12:57:25.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:25.153+0000 7ff3028c6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:25.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:25.169+0000 7ff3028c6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:26.015 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T12:57:26.041 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T12:57:26.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:26.153+0000 7f87c58d0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:26.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:26.169+0000 7f87c58d0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:27.026 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T12:57:27.055 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T12:57:27.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:27.169+0000 7f47d4fb8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:27.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:27.189+0000 7f47d4fb8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T12:57:28.066 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T12:57:28.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:28.185+0000 7f7faa8cd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:28.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:28.201+0000 7f7faa8cd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:29.051 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T12:57:29.079 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T12:57:29.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:29.201+0000 7f61cba99640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:29.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:29.217+0000 7f61cba99640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:30.067 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T12:57:30.092 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T12:57:30.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:30.201+0000 7fb778304640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:30.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:30.217+0000 7fb778304640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:31.087 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T12:57:31.115 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T12:57:31.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:31.233+0000 7f33d46d3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:31.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:31.249+0000 7f33d46d3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:32.090 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T12:57:32.119 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T12:57:32.119 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:57:32.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:32.233+0000 7f28e4538640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:32.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:32.245+0000 7f28e4538640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:32.427 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:57:32.428 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T12:57:32.446 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-03-30T12:57:32.092060+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":101,"num_read_kb":4622,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44773254,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":312936,"kb_used_data":96240,"kb_used_omap":15,"kb_used_meta":216560,"kb_avail":754661784,"statfs":{"total":773094113280,"available":772773666816,"internally_reserved":0,"allocated":98549760,"data_stored":95988457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15560,"internal_metadata":221758264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.047940"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":104,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-03-30T12:57:29.310218+0000","last_change":"2024-03-30T12:57:16.932276+0000","last_active":"2024-03-30T12:57:29.310218+0000","last_peered":"2024-03-30T12:57:29.310218+0000","last_clean":"2024-03-30T12:57:29.310218+0000","last_became_active":"2024-03-30T12:57:16.931067+0000","last_became_peered":"2024-03-30T12:57:16.931067+0000","last_unstale":"2024-03-30T12:57:29.310218+0000","last_undegraded":"2024-03-30T12:57:29.310218+0000","last_fullsized":"2024-03-30T12:57:29.310218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:12.143245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:12.143245+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:38.357453+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":58,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-30T12:57:30.071029+0000","last_change":"2024-03-30T12:57:16.932484+0000","last_active":"2024-03-30T12:57:30.071029+0000","last_peered":"2024-03-30T12:57:30.071029+0000","last_clean":"2024-03-30T12:57:30.071029+0000","last_became_active":"2024-03-30T12:57:16.931081+0000","last_became_peered":"2024-03-30T12:57:16.931081+0000","last_unstale":"2024-03-30T12:57:30.071029+0000","last_undegraded":"2024-03-30T12:57:30.071029+0000","last_fullsized":"2024-03-30T12:57:30.071029+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:56:54.949876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:54.949876+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:01.261626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":106,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-30T12:57:28.772529+0000","last_change":"2024-03-30T12:57:16.936123+0000","last_active":"2024-03-30T12:57:28.772529+0000","last_peered":"2024-03-30T12:57:28.772529+0000","last_clean":"2024-03-30T12:57:28.772529+0000","last_became_active":"2024-03-30T12:57:16.934982+0000","last_became_peered":"2024-03-30T12:57:16.934982+0000","last_unstale":"2024-03-30T12:57:28.772529+0000","last_undegraded":"2024-03-30T12:57:28.772529+0000","last_fullsized":"2024-03-30T12:57:28.772529+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:56:59.916447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:59.916447+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:17.560748+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":83,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-30T12:57:28.772459+0000","last_change":"2024-03-30T12:56:44.900397+0000","last_active":"2024-03-30T12:57:28.772459+0000","last_peered":"2024-03-30T12:57:28.772459+0000","last_clean":"2024-03-30T12:57:28.772459+0000","last_became_active":"2024-03-30T12:55:43.938430+0000","last_became_peered":"2024-03-30T12:55:43.938430+0000","last_unstale":"2024-03-30T12:57:28.772459+0000","last_undegraded":"2024-03-30T12:57:28.772459+0000","last_fullsized":"2024-03-30T12:57:28.772459+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:56:44.900077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:44.900077+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:04.497239+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":81,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-30T12:57:28.771966+0000","last_change":"2024-03-30T12:57:04.946767+0000","last_active":"2024-03-30T12:57:28.771966+0000","last_peered":"2024-03-30T12:57:28.771966+0000","last_clean":"2024-03-30T12:57:28.771966+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T12:57:28.771966+0000","last_undegraded":"2024-03-30T12:57:28.771966+0000","last_fullsized":"2024-03-30T12:57:28.771966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:57:04.946489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:04.946489+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:26.951070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-03-30T12:57:27.609950+0000","last_change":"2024-03-30T12:56:48.287112+0000","last_active":"2024-03-30T12:57:27.609950+0000","last_peered":"2024-03-30T12:57:27.609950+0000","last_clean":"2024-03-30T12:57:27.609950+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T12:57:27.609950+0000","last_undegraded":"2024-03-30T12:57:27.609950+0000","last_fullsized":"2024-03-30T12:57:27.609950+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:56:48.286816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:48.286816+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:59.500682+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":124,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-30T12:57:28.772480+0000","last_change":"2024-03-30T12:57:16.937077+0000","last_active":"2024-03-30T12:57:28.772480+0000","last_peered":"2024-03-30T12:57:28.772480+0000","last_clean":"2024-03-30T12:57:28.772480+0000","last_became_active":"2024-03-30T12:57:16.935779+0000","last_became_peered":"2024-03-30T12:57:16.935779+0000","last_unstale":"2024-03-30T12:57:28.772480+0000","last_undegraded":"2024-03-30T12:57:28.772480+0000","last_fullsized":"2024-03-30T12:57:28.772480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T12:57:02.954261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:02.954261+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:14.892580+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":91,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-03-30T12:57:29.057480+0000","last_change":"2024-03-30T12:56:58.446137+0000","last_active":"2024-03-30T12:57:29.057480+0000","last_peered":"2024-03-30T12:57:29.057480+0000","last_clean":"2024-03-30T12:57:29.057480+0000","last_became_active":"2024-03-30T12:55:43.930722+0000","last_became_peered":"2024-03-30T12:55:43.930722+0000","last_unstale":"2024-03-30T12:57:29.057480+0000","last_undegraded":"2024-03-30T12:57:29.057480+0000","last_fullsized":"2024-03-30T12:57:29.057480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:56:58.445737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:58.445737+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:21.311535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":82,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-03-30T12:57:27.610053+0000","last_change":"2024-03-30T12:57:00.268436+0000","last_active":"2024-03-30T12:57:27.610053+0000","last_peered":"2024-03-30T12:57:27.610053+0000","last_clean":"2024-03-30T12:57:27.610053+0000","last_became_active":"2024-03-30T12:55:43.932863+0000","last_became_peered":"2024-03-30T12:55:43.932863+0000","last_unstale":"2024-03-30T12:57:27.610053+0000","last_undegraded":"2024-03-30T12:57:27.610053+0000","last_fullsized":"2024-03-30T12:57:27.610053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:00.268171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:00.268171+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:11.652557+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":58,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-30T12:57:30.070931+0000","last_change":"2024-03-30T12:57:16.932042+0000","last_active":"2024-03-30T12:57:30.070931+0000","last_peered":"2024-03-30T12:57:30.070931+0000","last_clean":"2024-03-30T12:57:30.070931+0000","last_became_active":"2024-03-30T12:57:16.930714+0000","last_became_peered":"2024-03-30T12:57:16.930714+0000","last_unstale":"2024-03-30T12:57:30.070931+0000","last_undegraded":"2024-03-30T12:57:30.070931+0000","last_fullsized":"2024-03-30T12:57:30.070931+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:57:11.300486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:11.300486+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:29.376314+0000","scrub_duration":503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":125,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-03-30T12:57:29.310190+0000","last_change":"2024-03-30T12:57:16.932035+0000","last_active":"2024-03-30T12:57:29.310190+0000","last_peered":"2024-03-30T12:57:29.310190+0000","last_clean":"2024-03-30T12:57:29.310190+0000","last_became_active":"2024-03-30T12:57:16.930513+0000","last_became_peered":"2024-03-30T12:57:16.930513+0000","last_unstale":"2024-03-30T12:57:29.310190+0000","last_undegraded":"2024-03-30T12:57:29.310190+0000","last_fullsized":"2024-03-30T12:57:29.310190+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:57:00.039850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:00.039850+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:15.642502+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":110,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-03-30T12:57:29.310259+0000","last_change":"2024-03-30T12:57:11.126991+0000","last_active":"2024-03-30T12:57:29.310259+0000","last_peered":"2024-03-30T12:57:29.310259+0000","last_clean":"2024-03-30T12:57:29.310259+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T12:57:29.310259+0000","last_undegraded":"2024-03-30T12:57:29.310259+0000","last_fullsized":"2024-03-30T12:57:29.310259+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T12:57:11.126678+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:20.211259+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":93,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-30T12:57:28.771587+0000","last_change":"2024-03-30T12:56:56.068730+0000","last_active":"2024-03-30T12:57:28.771587+0000","last_peered":"2024-03-30T12:57:28.771587+0000","last_clean":"2024-03-30T12:57:28.771587+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T12:57:28.771587+0000","last_undegraded":"2024-03-30T12:57:28.771587+0000","last_fullsized":"2024-03-30T12:57:28.771587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:56:56.068479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:56.068479+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:08.832176+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":79,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-30T12:57:28.772480+0000","last_change":"2024-03-30T12:56:56.870958+0000","last_active":"2024-03-30T12:57:28.772480+0000","last_peered":"2024-03-30T12:57:28.772480+0000","last_clean":"2024-03-30T12:57:28.772480+0000","last_became_active":"2024-03-30T12:55:43.934495+0000","last_became_peered":"2024-03-30T12:55:43.934495+0000","last_unstale":"2024-03-30T12:57:28.772480+0000","last_undegraded":"2024-03-30T12:57:28.772480+0000","last_fullsized":"2024-03-30T12:57:28.772480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:56:56.870694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:56.870694+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:20.371042+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":93,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-30T12:57:28.771971+0000","last_change":"2024-03-30T12:56:50.034844+0000","last_active":"2024-03-30T12:57:28.771971+0000","last_peered":"2024-03-30T12:57:28.771971+0000","last_clean":"2024-03-30T12:57:28.771971+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T12:57:28.771971+0000","last_undegraded":"2024-03-30T12:57:28.771971+0000","last_fullsized":"2024-03-30T12:57:28.771971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:56:50.034645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:50.034645+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:07.024201+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":87,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-03-30T12:57:29.057309+0000","last_change":"2024-03-30T12:57:09.558059+0000","last_active":"2024-03-30T12:57:29.057309+0000","last_peered":"2024-03-30T12:57:29.057309+0000","last_clean":"2024-03-30T12:57:29.057309+0000","last_became_active":"2024-03-30T12:55:43.925945+0000","last_became_peered":"2024-03-30T12:55:43.925945+0000","last_unstale":"2024-03-30T12:57:29.057309+0000","last_undegraded":"2024-03-30T12:57:29.057309+0000","last_fullsized":"2024-03-30T12:57:29.057309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:09.557910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:09.557910+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:37.248333+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":101,"num_read_kb":4622,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44773254,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94236672,"data_stored":94236672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509466,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42664,"kb_used_data":15636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329176,"statfs":{"total":96636764160,"available":96593076224,"internally_reserved":0,"allocated":16011264,"data_stored":15690251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542170,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46168,"kb_used_data":19148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94325672,"statfs":{"total":96636764160,"available":96589488128,"internally_reserved":0,"allocated":19607552,"data_stored":19293340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822660,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39452,"kb_used_data":11920,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94332388,"statfs":{"total":96636764160,"available":96596365312,"internally_reserved":0,"allocated":12206080,"data_stored":11871465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4399,"internal_metadata":28176081},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509467,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33872,"kb_used_data":6852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337968,"statfs":{"total":96636764160,"available":96602079232,"internally_reserved":0,"allocated":7016448,"data_stored":6695435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509467,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32732,"kb_used_data":5708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339108,"statfs":{"total":96636764160,"available":96603246592,"internally_reserved":0,"allocated":5844992,"data_stored":5517178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509467,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41052,"kb_used_data":14028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330788,"statfs":{"total":96636764160,"available":96594726912,"internally_reserved":0,"allocated":14364672,"data_stored":14043659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542170,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36964,"kb_used_data":9944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334876,"statfs":{"total":96636764160,"available":96598913024,"internally_reserved":0,"allocated":10182656,"data_stored":9868444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574875,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40032,"kb_used_data":13004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331808,"statfs":{"total":96636764160,"available":96595771392,"internally_reserved":0,"allocated":13316096,"data_stored":13008685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11640832,"data_stored":11640832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19079168,"data_stored":19079168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9654272,"data_stored":9654272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:57:32.447 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:57:32.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:32.565+0000 7fd0bf461640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:32.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:32.581+0000 7fd0bf461640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:32.758 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:57:32.758 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T12:57:32.776 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-03-30T12:57:32.092060+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":101,"num_read_kb":4622,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44773254,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":312936,"kb_used_data":96240,"kb_used_omap":15,"kb_used_meta":216560,"kb_avail":754661784,"statfs":{"total":773094113280,"available":772773666816,"internally_reserved":0,"allocated":98549760,"data_stored":95988457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15560,"internal_metadata":221758264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.047940"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":104,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-03-30T12:57:29.310218+0000","last_change":"2024-03-30T12:57:16.932276+0000","last_active":"2024-03-30T12:57:29.310218+0000","last_peered":"2024-03-30T12:57:29.310218+0000","last_clean":"2024-03-30T12:57:29.310218+0000","last_became_active":"2024-03-30T12:57:16.931067+0000","last_became_peered":"2024-03-30T12:57:16.931067+0000","last_unstale":"2024-03-30T12:57:29.310218+0000","last_undegraded":"2024-03-30T12:57:29.310218+0000","last_fullsized":"2024-03-30T12:57:29.310218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:12.143245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:12.143245+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:38.357453+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":58,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-30T12:57:30.071029+0000","last_change":"2024-03-30T12:57:16.932484+0000","last_active":"2024-03-30T12:57:30.071029+0000","last_peered":"2024-03-30T12:57:30.071029+0000","last_clean":"2024-03-30T12:57:30.071029+0000","last_became_active":"2024-03-30T12:57:16.931081+0000","last_became_peered":"2024-03-30T12:57:16.931081+0000","last_unstale":"2024-03-30T12:57:30.071029+0000","last_undegraded":"2024-03-30T12:57:30.071029+0000","last_fullsized":"2024-03-30T12:57:30.071029+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:56:54.949876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:54.949876+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:01.261626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":106,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-30T12:57:28.772529+0000","last_change":"2024-03-30T12:57:16.936123+0000","last_active":"2024-03-30T12:57:28.772529+0000","last_peered":"2024-03-30T12:57:28.772529+0000","last_clean":"2024-03-30T12:57:28.772529+0000","last_became_active":"2024-03-30T12:57:16.934982+0000","last_became_peered":"2024-03-30T12:57:16.934982+0000","last_unstale":"2024-03-30T12:57:28.772529+0000","last_undegraded":"2024-03-30T12:57:28.772529+0000","last_fullsized":"2024-03-30T12:57:28.772529+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:56:59.916447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:59.916447+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:17.560748+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":83,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-30T12:57:28.772459+0000","last_change":"2024-03-30T12:56:44.900397+0000","last_active":"2024-03-30T12:57:28.772459+0000","last_peered":"2024-03-30T12:57:28.772459+0000","last_clean":"2024-03-30T12:57:28.772459+0000","last_became_active":"2024-03-30T12:55:43.938430+0000","last_became_peered":"2024-03-30T12:55:43.938430+0000","last_unstale":"2024-03-30T12:57:28.772459+0000","last_undegraded":"2024-03-30T12:57:28.772459+0000","last_fullsized":"2024-03-30T12:57:28.772459+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:56:44.900077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:44.900077+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:04.497239+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":81,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-30T12:57:28.771966+0000","last_change":"2024-03-30T12:57:04.946767+0000","last_active":"2024-03-30T12:57:28.771966+0000","last_peered":"2024-03-30T12:57:28.771966+0000","last_clean":"2024-03-30T12:57:28.771966+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T12:57:28.771966+0000","last_undegraded":"2024-03-30T12:57:28.771966+0000","last_fullsized":"2024-03-30T12:57:28.771966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:57:04.946489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:04.946489+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:26.951070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":94,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-03-30T12:57:27.609950+0000","last_change":"2024-03-30T12:56:48.287112+0000","last_active":"2024-03-30T12:57:27.609950+0000","last_peered":"2024-03-30T12:57:27.609950+0000","last_clean":"2024-03-30T12:57:27.609950+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T12:57:27.609950+0000","last_undegraded":"2024-03-30T12:57:27.609950+0000","last_fullsized":"2024-03-30T12:57:27.609950+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:56:48.286816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:48.286816+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:59.500682+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":124,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-30T12:57:28.772480+0000","last_change":"2024-03-30T12:57:16.937077+0000","last_active":"2024-03-30T12:57:28.772480+0000","last_peered":"2024-03-30T12:57:28.772480+0000","last_clean":"2024-03-30T12:57:28.772480+0000","last_became_active":"2024-03-30T12:57:16.935779+0000","last_became_peered":"2024-03-30T12:57:16.935779+0000","last_unstale":"2024-03-30T12:57:28.772480+0000","last_undegraded":"2024-03-30T12:57:28.772480+0000","last_fullsized":"2024-03-30T12:57:28.772480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T12:57:02.954261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:02.954261+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:14.892580+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":91,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-03-30T12:57:29.057480+0000","last_change":"2024-03-30T12:56:58.446137+0000","last_active":"2024-03-30T12:57:29.057480+0000","last_peered":"2024-03-30T12:57:29.057480+0000","last_clean":"2024-03-30T12:57:29.057480+0000","last_became_active":"2024-03-30T12:55:43.930722+0000","last_became_peered":"2024-03-30T12:55:43.930722+0000","last_unstale":"2024-03-30T12:57:29.057480+0000","last_undegraded":"2024-03-30T12:57:29.057480+0000","last_fullsized":"2024-03-30T12:57:29.057480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:56:58.445737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:58.445737+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:21.311535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":82,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-03-30T12:57:27.610053+0000","last_change":"2024-03-30T12:57:00.268436+0000","last_active":"2024-03-30T12:57:27.610053+0000","last_peered":"2024-03-30T12:57:27.610053+0000","last_clean":"2024-03-30T12:57:27.610053+0000","last_became_active":"2024-03-30T12:55:43.932863+0000","last_became_peered":"2024-03-30T12:55:43.932863+0000","last_unstale":"2024-03-30T12:57:27.610053+0000","last_undegraded":"2024-03-30T12:57:27.610053+0000","last_fullsized":"2024-03-30T12:57:27.610053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:00.268171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:00.268171+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:11.652557+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":58,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-30T12:57:30.070931+0000","last_change":"2024-03-30T12:57:16.932042+0000","last_active":"2024-03-30T12:57:30.070931+0000","last_peered":"2024-03-30T12:57:30.070931+0000","last_clean":"2024-03-30T12:57:30.070931+0000","last_became_active":"2024-03-30T12:57:16.930714+0000","last_became_peered":"2024-03-30T12:57:16.930714+0000","last_unstale":"2024-03-30T12:57:30.070931+0000","last_undegraded":"2024-03-30T12:57:30.070931+0000","last_fullsized":"2024-03-30T12:57:30.070931+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:57:11.300486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:11.300486+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:29.376314+0000","scrub_duration":503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":125,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-03-30T12:57:29.310190+0000","last_change":"2024-03-30T12:57:16.932035+0000","last_active":"2024-03-30T12:57:29.310190+0000","last_peered":"2024-03-30T12:57:29.310190+0000","last_clean":"2024-03-30T12:57:29.310190+0000","last_became_active":"2024-03-30T12:57:16.930513+0000","last_became_peered":"2024-03-30T12:57:16.930513+0000","last_unstale":"2024-03-30T12:57:29.310190+0000","last_undegraded":"2024-03-30T12:57:29.310190+0000","last_fullsized":"2024-03-30T12:57:29.310190+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:57:00.039850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:00.039850+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:15.642502+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":110,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-03-30T12:57:29.310259+0000","last_change":"2024-03-30T12:57:11.126991+0000","last_active":"2024-03-30T12:57:29.310259+0000","last_peered":"2024-03-30T12:57:29.310259+0000","last_clean":"2024-03-30T12:57:29.310259+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T12:57:29.310259+0000","last_undegraded":"2024-03-30T12:57:29.310259+0000","last_fullsized":"2024-03-30T12:57:29.310259+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T12:57:11.126678+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:20.211259+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":93,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-30T12:57:28.771587+0000","last_change":"2024-03-30T12:56:56.068730+0000","last_active":"2024-03-30T12:57:28.771587+0000","last_peered":"2024-03-30T12:57:28.771587+0000","last_clean":"2024-03-30T12:57:28.771587+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T12:57:28.771587+0000","last_undegraded":"2024-03-30T12:57:28.771587+0000","last_fullsized":"2024-03-30T12:57:28.771587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:56:56.068479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:56.068479+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:08.832176+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":79,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-30T12:57:28.772480+0000","last_change":"2024-03-30T12:56:56.870958+0000","last_active":"2024-03-30T12:57:28.772480+0000","last_peered":"2024-03-30T12:57:28.772480+0000","last_clean":"2024-03-30T12:57:28.772480+0000","last_became_active":"2024-03-30T12:55:43.934495+0000","last_became_peered":"2024-03-30T12:55:43.934495+0000","last_unstale":"2024-03-30T12:57:28.772480+0000","last_undegraded":"2024-03-30T12:57:28.772480+0000","last_fullsized":"2024-03-30T12:57:28.772480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:56:56.870694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:56.870694+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:20.371042+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":93,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-30T12:57:28.771971+0000","last_change":"2024-03-30T12:56:50.034844+0000","last_active":"2024-03-30T12:57:28.771971+0000","last_peered":"2024-03-30T12:57:28.771971+0000","last_clean":"2024-03-30T12:57:28.771971+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T12:57:28.771971+0000","last_undegraded":"2024-03-30T12:57:28.771971+0000","last_fullsized":"2024-03-30T12:57:28.771971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:56:50.034645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:50.034645+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:07.024201+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":87,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-03-30T12:57:29.057309+0000","last_change":"2024-03-30T12:57:09.558059+0000","last_active":"2024-03-30T12:57:29.057309+0000","last_peered":"2024-03-30T12:57:29.057309+0000","last_clean":"2024-03-30T12:57:29.057309+0000","last_became_active":"2024-03-30T12:55:43.925945+0000","last_became_peered":"2024-03-30T12:55:43.925945+0000","last_unstale":"2024-03-30T12:57:29.057309+0000","last_undegraded":"2024-03-30T12:57:29.057309+0000","last_fullsized":"2024-03-30T12:57:29.057309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:09.557910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:09.557910+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:37.248333+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":101,"num_read_kb":4622,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44773254,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94236672,"data_stored":94236672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509466,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42664,"kb_used_data":15636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329176,"statfs":{"total":96636764160,"available":96593076224,"internally_reserved":0,"allocated":16011264,"data_stored":15690251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542170,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46168,"kb_used_data":19148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94325672,"statfs":{"total":96636764160,"available":96589488128,"internally_reserved":0,"allocated":19607552,"data_stored":19293340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822660,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39452,"kb_used_data":11920,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94332388,"statfs":{"total":96636764160,"available":96596365312,"internally_reserved":0,"allocated":12206080,"data_stored":11871465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4399,"internal_metadata":28176081},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509467,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33872,"kb_used_data":6852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337968,"statfs":{"total":96636764160,"available":96602079232,"internally_reserved":0,"allocated":7016448,"data_stored":6695435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509467,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32732,"kb_used_data":5708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339108,"statfs":{"total":96636764160,"available":96603246592,"internally_reserved":0,"allocated":5844992,"data_stored":5517178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509467,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41052,"kb_used_data":14028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330788,"statfs":{"total":96636764160,"available":96594726912,"internally_reserved":0,"allocated":14364672,"data_stored":14043659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542170,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36964,"kb_used_data":9944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334876,"statfs":{"total":96636764160,"available":96598913024,"internally_reserved":0,"allocated":10182656,"data_stored":9868444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574875,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40032,"kb_used_data":13004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331808,"statfs":{"total":96636764160,"available":96595771392,"internally_reserved":0,"allocated":13316096,"data_stored":13008685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11640832,"data_stored":11640832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19079168,"data_stored":19079168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9654272,"data_stored":9654272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:57:32.777 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T12:57:32.777 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T12:57:32.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:32.893+0000 7f1a356e7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:32.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:32.905+0000 7f1a356e7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:33.079 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:57:33.079 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":52,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T12:57:32.087963+0000","last_up_change":"2024-03-30T12:57:15.896464+0000","last_in_change":"2024-03-30T12:57:15.896464+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3146012944},{"type":"v1","addr":"172.21.15.67:6818","nonce":3146012944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3146012944},{"type":"v1","addr":"172.21.15.67:6821","nonce":3146012944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3146012944},{"type":"v1","addr":"172.21.15.67:6828","nonce":3146012944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":3146012944},{"type":"v1","addr":"172.21.15.67:6825","nonce":3146012944}]},"public_addr":"172.21.15.67:6818/3146012944","cluster_addr":"172.21.15.67:6821/3146012944","heartbeat_back_addr":"172.21.15.67:6828/3146012944","heartbeat_front_addr":"172.21.15.67:6825/3146012944","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6801","nonce":1504348724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6803","nonce":1504348724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6807","nonce":1504348724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6805","nonce":1504348724}]},"public_addr":"172.21.15.67:6801/1504348724","cluster_addr":"172.21.15.67:6803/1504348724","heartbeat_back_addr":"172.21.15.67:6807/1504348724","heartbeat_front_addr":"172.21.15.67:6805/1504348724","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":25,"up_from":36,"up_thru":36,"down_at":26,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1927174653},{"type":"v1","addr":"172.21.15.82:6803","nonce":1927174653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":1927174653},{"type":"v1","addr":"172.21.15.82:6807","nonce":1927174653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":1927174653},{"type":"v1","addr":"172.21.15.82:6815","nonce":1927174653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":1927174653},{"type":"v1","addr":"172.21.15.82:6811","nonce":1927174653}]},"public_addr":"172.21.15.82:6803/1927174653","cluster_addr":"172.21.15.82:6807/1927174653","heartbeat_back_addr":"172.21.15.82:6815/1927174653","heartbeat_front_addr":"172.21.15.82:6811/1927174653","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6817","nonce":3467856022}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6819","nonce":3467856022}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6823","nonce":3467856022}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6821","nonce":3467856022}]},"public_addr":"172.21.15.82:6817/3467856022","cluster_addr":"172.21.15.82:6819/3467856022","heartbeat_back_addr":"172.21.15.82:6823/3467856022","heartbeat_front_addr":"172.21.15.82:6821/3467856022","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6801","nonce":968508548}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6806","nonce":968508548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6814","nonce":968508548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6810","nonce":968508548}]},"public_addr":"172.21.15.82:6801/968508548","cluster_addr":"172.21.15.82:6806/968508548","heartbeat_back_addr":"172.21.15.82:6814/968508548","heartbeat_front_addr":"172.21.15.82:6810/968508548","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2471694698},{"type":"v1","addr":"172.21.15.82:6825","nonce":2471694698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2471694698},{"type":"v1","addr":"172.21.15.82:6827","nonce":2471694698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2471694698},{"type":"v1","addr":"172.21.15.82:6831","nonce":2471694698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2471694698},{"type":"v1","addr":"172.21.15.82:6829","nonce":2471694698}]},"public_addr":"172.21.15.82:6825/2471694698","cluster_addr":"172.21.15.82:6827/2471694698","heartbeat_back_addr":"172.21.15.82:6831/2471694698","heartbeat_front_addr":"172.21.15.82:6829/2471694698","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:56:49.606349+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:57:00.846348+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:57:00.413778+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:56:52.677338+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:57:15.131405+0000","dead_epoch":26},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:56:54.119545+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:56:45.442378+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:56:47.788857+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T12:57:33.096 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T12:57:33.097 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T12:57:33.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:33.205+0000 7f9773185640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:33.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:33.221+0000 7f9773185640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:33.389 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T12:57:33.407 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T12:57:33.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:33.521+0000 7f17ed40e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:33.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:33.541+0000 7f17ed40e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:33.718 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:57:33.718 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T12:57:33.736 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T12:57:33.736 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T12:57:33.736 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T12:57:33.736 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T12:57:33.736 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 1, 2, 3, 5, 6, 7, 4] 2024-03-30T12:57:33.736 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T12:57:33.737 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:57:33.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:33.841+0000 7fd13837d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:33.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:33.853+0000 7fd13837d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:34.029 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:57:34.030 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T12:57:34.046 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2024-03-30T12:57:33.815144+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":101,"num_read_kb":4622,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44773254,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":312992,"kb_used_data":96300,"kb_used_omap":15,"kb_used_meta":216560,"kb_avail":754661728,"statfs":{"total":773094113280,"available":772773609472,"internally_reserved":0,"allocated":98611200,"data_stored":96015892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15560,"internal_metadata":221758264},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.762234"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":104,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-03-30T12:57:29.310218+0000","last_change":"2024-03-30T12:57:16.932276+0000","last_active":"2024-03-30T12:57:29.310218+0000","last_peered":"2024-03-30T12:57:29.310218+0000","last_clean":"2024-03-30T12:57:29.310218+0000","last_became_active":"2024-03-30T12:57:16.931067+0000","last_became_peered":"2024-03-30T12:57:16.931067+0000","last_unstale":"2024-03-30T12:57:29.310218+0000","last_undegraded":"2024-03-30T12:57:29.310218+0000","last_fullsized":"2024-03-30T12:57:29.310218+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:12.143245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:12.143245+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:38.357453+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":58,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-30T12:57:30.071029+0000","last_change":"2024-03-30T12:57:16.932484+0000","last_active":"2024-03-30T12:57:30.071029+0000","last_peered":"2024-03-30T12:57:30.071029+0000","last_clean":"2024-03-30T12:57:30.071029+0000","last_became_active":"2024-03-30T12:57:16.931081+0000","last_became_peered":"2024-03-30T12:57:16.931081+0000","last_unstale":"2024-03-30T12:57:30.071029+0000","last_undegraded":"2024-03-30T12:57:30.071029+0000","last_fullsized":"2024-03-30T12:57:30.071029+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:56:54.949876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:54.949876+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:01.261626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":106,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-30T12:57:28.772529+0000","last_change":"2024-03-30T12:57:16.936123+0000","last_active":"2024-03-30T12:57:28.772529+0000","last_peered":"2024-03-30T12:57:28.772529+0000","last_clean":"2024-03-30T12:57:28.772529+0000","last_became_active":"2024-03-30T12:57:16.934982+0000","last_became_peered":"2024-03-30T12:57:16.934982+0000","last_unstale":"2024-03-30T12:57:28.772529+0000","last_undegraded":"2024-03-30T12:57:28.772529+0000","last_fullsized":"2024-03-30T12:57:28.772529+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:56:59.916447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:59.916447+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:17.560748+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":83,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-30T12:57:28.772459+0000","last_change":"2024-03-30T12:56:44.900397+0000","last_active":"2024-03-30T12:57:28.772459+0000","last_peered":"2024-03-30T12:57:28.772459+0000","last_clean":"2024-03-30T12:57:28.772459+0000","last_became_active":"2024-03-30T12:55:43.938430+0000","last_became_peered":"2024-03-30T12:55:43.938430+0000","last_unstale":"2024-03-30T12:57:28.772459+0000","last_undegraded":"2024-03-30T12:57:28.772459+0000","last_fullsized":"2024-03-30T12:57:28.772459+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:56:44.900077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:44.900077+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:04.497239+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":81,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-30T12:57:28.771966+0000","last_change":"2024-03-30T12:57:04.946767+0000","last_active":"2024-03-30T12:57:28.771966+0000","last_peered":"2024-03-30T12:57:28.771966+0000","last_clean":"2024-03-30T12:57:28.771966+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T12:57:28.771966+0000","last_undegraded":"2024-03-30T12:57:28.771966+0000","last_fullsized":"2024-03-30T12:57:28.771966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:57:04.946489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:04.946489+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:26.951070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-30T12:57:32.489798+0000","last_change":"2024-03-30T12:56:48.287112+0000","last_active":"2024-03-30T12:57:32.489798+0000","last_peered":"2024-03-30T12:57:32.489798+0000","last_clean":"2024-03-30T12:57:32.489798+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T12:57:32.489798+0000","last_undegraded":"2024-03-30T12:57:32.489798+0000","last_fullsized":"2024-03-30T12:57:32.489798+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:56:48.286816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:48.286816+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:59.500682+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":124,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-30T12:57:28.772480+0000","last_change":"2024-03-30T12:57:16.937077+0000","last_active":"2024-03-30T12:57:28.772480+0000","last_peered":"2024-03-30T12:57:28.772480+0000","last_clean":"2024-03-30T12:57:28.772480+0000","last_became_active":"2024-03-30T12:57:16.935779+0000","last_became_peered":"2024-03-30T12:57:16.935779+0000","last_unstale":"2024-03-30T12:57:28.772480+0000","last_undegraded":"2024-03-30T12:57:28.772480+0000","last_fullsized":"2024-03-30T12:57:28.772480+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T12:57:02.954261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:02.954261+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:14.892580+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":91,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-03-30T12:57:29.057480+0000","last_change":"2024-03-30T12:56:58.446137+0000","last_active":"2024-03-30T12:57:29.057480+0000","last_peered":"2024-03-30T12:57:29.057480+0000","last_clean":"2024-03-30T12:57:29.057480+0000","last_became_active":"2024-03-30T12:55:43.930722+0000","last_became_peered":"2024-03-30T12:55:43.930722+0000","last_unstale":"2024-03-30T12:57:29.057480+0000","last_undegraded":"2024-03-30T12:57:29.057480+0000","last_fullsized":"2024-03-30T12:57:29.057480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:56:58.445737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:58.445737+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:21.311535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-30T12:57:32.489770+0000","last_change":"2024-03-30T12:57:00.268436+0000","last_active":"2024-03-30T12:57:32.489770+0000","last_peered":"2024-03-30T12:57:32.489770+0000","last_clean":"2024-03-30T12:57:32.489770+0000","last_became_active":"2024-03-30T12:55:43.932863+0000","last_became_peered":"2024-03-30T12:55:43.932863+0000","last_unstale":"2024-03-30T12:57:32.489770+0000","last_undegraded":"2024-03-30T12:57:32.489770+0000","last_fullsized":"2024-03-30T12:57:32.489770+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:00.268171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:00.268171+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:11.652557+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":58,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-30T12:57:30.070931+0000","last_change":"2024-03-30T12:57:16.932042+0000","last_active":"2024-03-30T12:57:30.070931+0000","last_peered":"2024-03-30T12:57:30.070931+0000","last_clean":"2024-03-30T12:57:30.070931+0000","last_became_active":"2024-03-30T12:57:16.930714+0000","last_became_peered":"2024-03-30T12:57:16.930714+0000","last_unstale":"2024-03-30T12:57:30.070931+0000","last_undegraded":"2024-03-30T12:57:30.070931+0000","last_fullsized":"2024-03-30T12:57:30.070931+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:57:11.300486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:11.300486+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:29.376314+0000","scrub_duration":503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":125,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-03-30T12:57:29.310190+0000","last_change":"2024-03-30T12:57:16.932035+0000","last_active":"2024-03-30T12:57:29.310190+0000","last_peered":"2024-03-30T12:57:29.310190+0000","last_clean":"2024-03-30T12:57:29.310190+0000","last_became_active":"2024-03-30T12:57:16.930513+0000","last_became_peered":"2024-03-30T12:57:16.930513+0000","last_unstale":"2024-03-30T12:57:29.310190+0000","last_undegraded":"2024-03-30T12:57:29.310190+0000","last_fullsized":"2024-03-30T12:57:29.310190+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:57:00.039850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:00.039850+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:15.642502+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":110,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-03-30T12:57:29.310259+0000","last_change":"2024-03-30T12:57:11.126991+0000","last_active":"2024-03-30T12:57:29.310259+0000","last_peered":"2024-03-30T12:57:29.310259+0000","last_clean":"2024-03-30T12:57:29.310259+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T12:57:29.310259+0000","last_undegraded":"2024-03-30T12:57:29.310259+0000","last_fullsized":"2024-03-30T12:57:29.310259+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T12:57:11.126678+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:20.211259+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":93,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-30T12:57:28.771587+0000","last_change":"2024-03-30T12:56:56.068730+0000","last_active":"2024-03-30T12:57:28.771587+0000","last_peered":"2024-03-30T12:57:28.771587+0000","last_clean":"2024-03-30T12:57:28.771587+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T12:57:28.771587+0000","last_undegraded":"2024-03-30T12:57:28.771587+0000","last_fullsized":"2024-03-30T12:57:28.771587+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:56:56.068479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:56.068479+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:08.832176+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":79,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-30T12:57:28.772480+0000","last_change":"2024-03-30T12:56:56.870958+0000","last_active":"2024-03-30T12:57:28.772480+0000","last_peered":"2024-03-30T12:57:28.772480+0000","last_clean":"2024-03-30T12:57:28.772480+0000","last_became_active":"2024-03-30T12:55:43.934495+0000","last_became_peered":"2024-03-30T12:55:43.934495+0000","last_unstale":"2024-03-30T12:57:28.772480+0000","last_undegraded":"2024-03-30T12:57:28.772480+0000","last_fullsized":"2024-03-30T12:57:28.772480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:56:56.870694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:56.870694+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:20.371042+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":93,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-30T12:57:28.771971+0000","last_change":"2024-03-30T12:56:50.034844+0000","last_active":"2024-03-30T12:57:28.771971+0000","last_peered":"2024-03-30T12:57:28.771971+0000","last_clean":"2024-03-30T12:57:28.771971+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T12:57:28.771971+0000","last_undegraded":"2024-03-30T12:57:28.771971+0000","last_fullsized":"2024-03-30T12:57:28.771971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:56:50.034645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:50.034645+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:07.024201+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":87,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-03-30T12:57:29.057309+0000","last_change":"2024-03-30T12:57:09.558059+0000","last_active":"2024-03-30T12:57:29.057309+0000","last_peered":"2024-03-30T12:57:29.057309+0000","last_clean":"2024-03-30T12:57:29.057309+0000","last_became_active":"2024-03-30T12:55:43.925945+0000","last_became_peered":"2024-03-30T12:55:43.925945+0000","last_unstale":"2024-03-30T12:57:29.057309+0000","last_undegraded":"2024-03-30T12:57:29.057309+0000","last_fullsized":"2024-03-30T12:57:29.057309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:09.557910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:09.557910+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:37.248333+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":101,"num_read_kb":4622,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44773254,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94236672,"data_stored":94236672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509466,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42664,"kb_used_data":15636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329176,"statfs":{"total":96636764160,"available":96593076224,"internally_reserved":0,"allocated":16011264,"data_stored":15690251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542170,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46168,"kb_used_data":19148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94325672,"statfs":{"total":96636764160,"available":96589488128,"internally_reserved":0,"allocated":19607552,"data_stored":19293340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822660,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39452,"kb_used_data":11920,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94332388,"statfs":{"total":96636764160,"available":96596365312,"internally_reserved":0,"allocated":12206080,"data_stored":11871465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4399,"internal_metadata":28176081},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509467,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33872,"kb_used_data":6852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337968,"statfs":{"total":96636764160,"available":96602079232,"internally_reserved":0,"allocated":7016448,"data_stored":6695435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509467,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32732,"kb_used_data":5708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339108,"statfs":{"total":96636764160,"available":96603246592,"internally_reserved":0,"allocated":5844992,"data_stored":5517178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509467,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41052,"kb_used_data":14028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330788,"statfs":{"total":96636764160,"available":96594726912,"internally_reserved":0,"allocated":14364672,"data_stored":14043659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":14,"seq":60129542170,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36964,"kb_used_data":9944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334876,"statfs":{"total":96636764160,"available":96598913024,"internally_reserved":0,"allocated":10182656,"data_stored":9868444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40088,"kb_used_data":13064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331752,"statfs":{"total":96636764160,"available":96595714048,"internally_reserved":0,"allocated":13377536,"data_stored":13036120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11640832,"data_stored":11640832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19079168,"data_stored":19079168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9654272,"data_stored":9654272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:57:34.048 INFO:tasks.thrashosds.thrasher:Choosing PG 1.0 with acting set [7, 0, 6] 2024-03-30T12:57:34.048 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 1, 2, 3, 5, 6, 7, 4] 2024-03-30T12:57:34.048 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-30T12:57:34.048 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T12:57:34.108 INFO:tasks.ceph.osd.7:Stopped 2024-03-30T12:57:34.108 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 2, 3, 5, 6, 7, 4] 2024-03-30T12:57:34.108 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 7 2024-03-30T12:57:34.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:34.209+0000 7fc7fb608640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:34.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:34.221+0000 7fc7fb608640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:34.399 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T12:57:34.397+0000 7fbb38f08640 -1 mon.a@0(leader).osd e53 definitely_dead 0 2024-03-30T12:57:34.613 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-03-30T12:57:35.110 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.7. 2024-03-30T12:57:35.141 INFO:tasks.thrashosds.thrasher:dead_osds=[7], live_osds=[0, 1, 2, 3, 5, 6, 4] 2024-03-30T12:57:35.141 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T12:57:35.141 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:57:35.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:35.253+0000 7fa3e4509640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:35.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:57:35.265+0000 7fa3e4509640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:57:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:57:35.436 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T12:57:35.452 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2024-03-30T12:57:35.111679+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":99,"num_read_kb":4620,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":30134555,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":276304,"kb_used_data":86632,"kb_used_omap":13,"kb_used_meta":189554,"kb_avail":660326576,"statfs":{"total":676457349120,"available":676174413824,"internally_reserved":0,"allocated":88711168,"data_stored":86274177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13970,"internal_metadata":194103662},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-2,"num_read_kb":-2,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.042978"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":112,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-30T12:57:34.114248+0000","last_change":"2024-03-30T12:57:16.932276+0000","last_active":"2024-03-30T12:57:34.114248+0000","last_peered":"2024-03-30T12:57:34.114248+0000","last_clean":"2024-03-30T12:57:34.114248+0000","last_became_active":"2024-03-30T12:57:16.931067+0000","last_became_peered":"2024-03-30T12:57:16.931067+0000","last_unstale":"2024-03-30T12:57:34.114248+0000","last_undegraded":"2024-03-30T12:57:34.114248+0000","last_fullsized":"2024-03-30T12:57:34.114248+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:12.143245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:12.143245+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:38.357453+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":58,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-30T12:57:30.071029+0000","last_change":"2024-03-30T12:57:16.932484+0000","last_active":"2024-03-30T12:57:30.071029+0000","last_peered":"2024-03-30T12:57:30.071029+0000","last_clean":"2024-03-30T12:57:30.071029+0000","last_became_active":"2024-03-30T12:57:16.931081+0000","last_became_peered":"2024-03-30T12:57:16.931081+0000","last_unstale":"2024-03-30T12:57:30.071029+0000","last_undegraded":"2024-03-30T12:57:30.071029+0000","last_fullsized":"2024-03-30T12:57:30.071029+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:56:54.949876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:54.949876+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:01.261626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":114,"reported_epoch":52,"state":"stale+active+clean","last_fresh":"2024-03-30T12:57:32.516232+0000","last_change":"2024-03-30T12:57:16.936123+0000","last_active":"2024-03-30T12:57:32.516232+0000","last_peered":"2024-03-30T12:57:32.516232+0000","last_clean":"2024-03-30T12:57:32.516232+0000","last_became_active":"2024-03-30T12:57:16.934982+0000","last_became_peered":"2024-03-30T12:57:16.934982+0000","last_unstale":"2024-03-30T12:57:34.111615+0000","last_undegraded":"2024-03-30T12:57:32.516232+0000","last_fullsized":"2024-03-30T12:57:32.516232+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:56:59.916447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:59.916447+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:17.560748+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":25,"reported_epoch":53,"state":"peering","last_fresh":"2024-03-30T12:57:34.111109+0000","last_change":"2024-03-30T12:57:34.111109+0000","last_active":"2024-03-30T12:57:34.111109+0000","last_peered":"2024-03-30T12:55:47.313349+0000","last_clean":"2024-03-30T12:55:47.313349+0000","last_became_active":"2024-03-30T12:55:43.938430+0000","last_became_peered":"2024-03-30T12:55:43.938430+0000","last_unstale":"2024-03-30T12:57:34.111109+0000","last_undegraded":"2024-03-30T12:57:34.111109+0000","last_fullsized":"2024-03-30T12:57:34.111109+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:56:44.900077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:44.900077+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,6],"acting":[2147483647,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":89,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-30T12:57:32.096847+0000","last_change":"2024-03-30T12:57:04.946767+0000","last_active":"2024-03-30T12:57:32.096847+0000","last_peered":"2024-03-30T12:57:32.096847+0000","last_clean":"2024-03-30T12:57:32.096847+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T12:57:32.096847+0000","last_undegraded":"2024-03-30T12:57:32.096847+0000","last_fullsized":"2024-03-30T12:57:32.096847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:57:04.946489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:04.946489+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:26.951070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":104,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-30T12:57:32.489798+0000","last_change":"2024-03-30T12:56:48.287112+0000","last_active":"2024-03-30T12:57:32.489798+0000","last_peered":"2024-03-30T12:57:32.489798+0000","last_clean":"2024-03-30T12:57:32.489798+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T12:57:32.489798+0000","last_undegraded":"2024-03-30T12:57:32.489798+0000","last_fullsized":"2024-03-30T12:57:32.489798+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:56:48.286816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:48.286816+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:59.500682+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":41,"reported_epoch":53,"state":"peering","last_fresh":"2024-03-30T12:57:34.111127+0000","last_change":"2024-03-30T12:57:34.111127+0000","last_active":"2024-03-30T12:57:34.111127+0000","last_peered":"2024-03-30T12:55:47.587476+0000","last_clean":"2024-03-30T12:55:47.587476+0000","last_became_active":"2024-03-30T12:55:43.933833+0000","last_became_peered":"2024-03-30T12:55:43.933833+0000","last_unstale":"2024-03-30T12:57:34.111127+0000","last_undegraded":"2024-03-30T12:57:34.111127+0000","last_fullsized":"2024-03-30T12:57:34.111127+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T12:57:02.954261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:02.954261+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,4],"acting":[2147483647,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":99,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-30T12:57:34.110570+0000","last_change":"2024-03-30T12:56:58.446137+0000","last_active":"2024-03-30T12:57:34.110570+0000","last_peered":"2024-03-30T12:57:34.110570+0000","last_clean":"2024-03-30T12:57:34.110570+0000","last_became_active":"2024-03-30T12:55:43.930722+0000","last_became_peered":"2024-03-30T12:55:43.930722+0000","last_unstale":"2024-03-30T12:57:34.110570+0000","last_undegraded":"2024-03-30T12:57:34.110570+0000","last_fullsized":"2024-03-30T12:57:34.110570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:56:58.445737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:58.445737+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:21.311535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":92,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-30T12:57:32.489770+0000","last_change":"2024-03-30T12:57:00.268436+0000","last_active":"2024-03-30T12:57:32.489770+0000","last_peered":"2024-03-30T12:57:32.489770+0000","last_clean":"2024-03-30T12:57:32.489770+0000","last_became_active":"2024-03-30T12:55:43.932863+0000","last_became_peered":"2024-03-30T12:55:43.932863+0000","last_unstale":"2024-03-30T12:57:32.489770+0000","last_undegraded":"2024-03-30T12:57:32.489770+0000","last_fullsized":"2024-03-30T12:57:32.489770+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:00.268171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:00.268171+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:11.652557+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":58,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-03-30T12:57:30.070931+0000","last_change":"2024-03-30T12:57:16.932042+0000","last_active":"2024-03-30T12:57:30.070931+0000","last_peered":"2024-03-30T12:57:30.070931+0000","last_clean":"2024-03-30T12:57:30.070931+0000","last_became_active":"2024-03-30T12:57:16.930714+0000","last_became_peered":"2024-03-30T12:57:16.930714+0000","last_unstale":"2024-03-30T12:57:30.070931+0000","last_undegraded":"2024-03-30T12:57:30.070931+0000","last_fullsized":"2024-03-30T12:57:30.070931+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:57:11.300486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:11.300486+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:29.376314+0000","scrub_duration":503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":133,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-30T12:57:34.114105+0000","last_change":"2024-03-30T12:57:16.932035+0000","last_active":"2024-03-30T12:57:34.114105+0000","last_peered":"2024-03-30T12:57:34.114105+0000","last_clean":"2024-03-30T12:57:34.114105+0000","last_became_active":"2024-03-30T12:57:16.930513+0000","last_became_peered":"2024-03-30T12:57:16.930513+0000","last_unstale":"2024-03-30T12:57:34.114105+0000","last_undegraded":"2024-03-30T12:57:34.114105+0000","last_fullsized":"2024-03-30T12:57:34.114105+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:57:00.039850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:00.039850+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:15.642502+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":118,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-30T12:57:34.114399+0000","last_change":"2024-03-30T12:57:11.126991+0000","last_active":"2024-03-30T12:57:34.114399+0000","last_peered":"2024-03-30T12:57:34.114399+0000","last_clean":"2024-03-30T12:57:34.114399+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T12:57:34.114399+0000","last_undegraded":"2024-03-30T12:57:34.114399+0000","last_fullsized":"2024-03-30T12:57:34.114399+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T12:57:11.126678+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:20.211259+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":101,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-30T12:57:32.096816+0000","last_change":"2024-03-30T12:56:56.068730+0000","last_active":"2024-03-30T12:57:32.096816+0000","last_peered":"2024-03-30T12:57:32.096816+0000","last_clean":"2024-03-30T12:57:32.096816+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T12:57:32.096816+0000","last_undegraded":"2024-03-30T12:57:32.096816+0000","last_fullsized":"2024-03-30T12:57:32.096816+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:56:56.068479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:56.068479+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:08.832176+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":87,"reported_epoch":52,"state":"stale+active+clean","last_fresh":"2024-03-30T12:57:32.516141+0000","last_change":"2024-03-30T12:56:56.870958+0000","last_active":"2024-03-30T12:57:32.516141+0000","last_peered":"2024-03-30T12:57:32.516141+0000","last_clean":"2024-03-30T12:57:32.516141+0000","last_became_active":"2024-03-30T12:55:43.934495+0000","last_became_peered":"2024-03-30T12:55:43.934495+0000","last_unstale":"2024-03-30T12:57:34.111619+0000","last_undegraded":"2024-03-30T12:57:32.516141+0000","last_fullsized":"2024-03-30T12:57:32.516141+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:56:56.870694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:56.870694+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:20.371042+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":101,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-03-30T12:57:32.096864+0000","last_change":"2024-03-30T12:56:50.034844+0000","last_active":"2024-03-30T12:57:32.096864+0000","last_peered":"2024-03-30T12:57:32.096864+0000","last_clean":"2024-03-30T12:57:32.096864+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T12:57:32.096864+0000","last_undegraded":"2024-03-30T12:57:32.096864+0000","last_fullsized":"2024-03-30T12:57:32.096864+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:56:50.034645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:50.034645+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:07.024201+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":95,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-30T12:57:34.110441+0000","last_change":"2024-03-30T12:57:09.558059+0000","last_active":"2024-03-30T12:57:34.110441+0000","last_peered":"2024-03-30T12:57:34.110441+0000","last_clean":"2024-03-30T12:57:34.110441+0000","last_became_active":"2024-03-30T12:55:43.925945+0000","last_became_peered":"2024-03-30T12:55:43.925945+0000","last_unstale":"2024-03-30T12:57:34.110441+0000","last_undegraded":"2024-03-30T12:57:34.110441+0000","last_fullsized":"2024-03-30T12:57:34.110441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:09.557910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:09.557910+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:37.248333+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":99,"num_read_kb":4620,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":30134555,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94236672,"data_stored":94236672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509467,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42720,"kb_used_data":15696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329120,"statfs":{"total":96636764160,"available":96593018880,"internally_reserved":0,"allocated":16072704,"data_stored":15717862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542171,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46232,"kb_used_data":19208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94325608,"statfs":{"total":96636764160,"available":96589422592,"internally_reserved":0,"allocated":19668992,"data_stored":19320775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822660,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39452,"kb_used_data":11920,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94332388,"statfs":{"total":96636764160,"available":96596365312,"internally_reserved":0,"allocated":12206080,"data_stored":11871465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4399,"internal_metadata":28176081},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509468,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33932,"kb_used_data":6912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337908,"statfs":{"total":96636764160,"available":96602017792,"internally_reserved":0,"allocated":7077888,"data_stored":6723046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509468,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32776,"kb_used_data":5756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339064,"statfs":{"total":96636764160,"available":96603201536,"internally_reserved":0,"allocated":5894144,"data_stored":5539302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509468,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41104,"kb_used_data":14076,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330736,"statfs":{"total":96636764160,"available":96594673664,"internally_reserved":0,"allocated":14413824,"data_stored":14065607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40088,"kb_used_data":13064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331752,"statfs":{"total":96636764160,"available":96595714048,"internally_reserved":0,"allocated":13377536,"data_stored":13036120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11640832,"data_stored":11640832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19079168,"data_stored":19079168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9654272,"data_stored":9654272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:57:35.453 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active or peered 2024-03-30T12:57:35.453 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "22'6", 'reported_seq': 25, 'reported_epoch': 53, 'state': 'peering', 'last_fresh': '2024-03-30T12:57:34.111109+0000', 'last_change': '2024-03-30T12:57:34.111109+0000', 'last_active': '2024-03-30T12:57:34.111109+0000', 'last_peered': '2024-03-30T12:55:47.313349+0000', 'last_clean': '2024-03-30T12:55:47.313349+0000', 'last_became_active': '2024-03-30T12:55:43.938430+0000', 'last_became_peered': '2024-03-30T12:55:43.938430+0000', 'last_unstale': '2024-03-30T12:57:34.111109+0000', 'last_undegraded': '2024-03-30T12:57:34.111109+0000', 'last_fullsized': '2024-03-30T12:57:34.111109+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 20, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "22'6", 'last_scrub_stamp': '2024-03-30T12:56:44.900077+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-30T12:55:42.900074+0000', 'last_clean_scrub_stamp': '2024-03-30T12:56:44.900077+0000', 'objects_scrubbed': 0, 'log_size': 6, 'log_dups_size': 0, 'ondisk_log_size': 6, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4588245, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 3, 'num_read_kb': 1, 'num_write': 8, 'num_write_kb': 2137, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 0, 6], 'acting': [2147483647, 0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-30T12:57:35.454 INFO:tasks.ceph.ceph_manager.ceph:PG 1.2 is not active or peered 2024-03-30T12:57:35.454 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.2', 'version': "22'18", 'reported_seq': 41, 'reported_epoch': 53, 'state': 'peering', 'last_fresh': '2024-03-30T12:57:34.111127+0000', 'last_change': '2024-03-30T12:57:34.111127+0000', 'last_active': '2024-03-30T12:57:34.111127+0000', 'last_peered': '2024-03-30T12:55:47.587476+0000', 'last_clean': '2024-03-30T12:55:47.587476+0000', 'last_became_active': '2024-03-30T12:55:43.933833+0000', 'last_became_peered': '2024-03-30T12:55:43.933833+0000', 'last_unstale': '2024-03-30T12:57:34.111127+0000', 'last_undegraded': '2024-03-30T12:57:34.111127+0000', 'last_fullsized': '2024-03-30T12:57:34.111127+0000', 'mapping_epoch': 53, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 37, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "22'18", 'last_scrub_stamp': '2024-03-30T12:57:02.954261+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-30T12:55:42.900074+0000', 'last_clean_scrub_stamp': '2024-03-30T12:57:02.954261+0000', 'objects_scrubbed': 0, 'log_size': 18, 'log_dups_size': 0, 'ondisk_log_size': 18, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14638699, 'num_objects': 6, 'num_object_clones': 0, 'num_object_copies': 18, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6, 'num_whiteouts': 0, 'num_read': 11, 'num_read_kb': 5, 'num_write': 24, 'num_write_kb': 6991, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 0, 4], 'acting': [2147483647, 0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-30T12:57:35.454 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T12:57:40.822 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-03-30T12:58:00.455 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T12:58:00.455 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:58:00.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:00.580+0000 7f09bf074640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:00.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:00.592+0000 7f09bf074640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:00.768 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:58:00.769 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T12:58:00.785 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":123,"stamp":"2024-03-30T12:57:59.822441+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":97,"num_read_kb":4618,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":59580023,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":286852,"kb_used_data":97200,"kb_used_omap":13,"kb_used_meta":189554,"kb_avail":660316028,"statfs":{"total":676457349120,"available":676163612672,"internally_reserved":0,"allocated":99532800,"data_stored":96907121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13970,"internal_metadata":194103662},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002787"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":118,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-30T12:57:52.323348+0000","last_change":"2024-03-30T12:57:16.932276+0000","last_active":"2024-03-30T12:57:52.323348+0000","last_peered":"2024-03-30T12:57:52.323348+0000","last_clean":"2024-03-30T12:57:52.323348+0000","last_became_active":"2024-03-30T12:57:16.931067+0000","last_became_peered":"2024-03-30T12:57:16.931067+0000","last_unstale":"2024-03-30T12:57:52.323348+0000","last_undegraded":"2024-03-30T12:57:52.323348+0000","last_fullsized":"2024-03-30T12:57:52.323348+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:12.143245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:12.143245+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:38.357453+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":79,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-30T12:57:52.234113+0000","last_change":"2024-03-30T12:57:36.810598+0000","last_active":"2024-03-30T12:57:52.234113+0000","last_peered":"2024-03-30T12:57:52.234113+0000","last_clean":"2024-03-30T12:57:52.234113+0000","last_became_active":"2024-03-30T12:57:36.132383+0000","last_became_peered":"2024-03-30T12:57:36.132383+0000","last_unstale":"2024-03-30T12:57:52.234113+0000","last_undegraded":"2024-03-30T12:57:52.234113+0000","last_fullsized":"2024-03-30T12:57:52.234113+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:56:54.949876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:54.949876+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:18.589705+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":21,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-30T12:57:52.324312+0000","last_change":"2024-03-30T12:57:36.209894+0000","last_active":"2024-03-30T12:57:52.324312+0000","last_peered":"2024-03-30T12:57:52.324312+0000","last_clean":"2024-03-30T12:57:52.324312+0000","last_became_active":"2024-03-30T12:57:36.132560+0000","last_became_peered":"2024-03-30T12:57:36.132560+0000","last_unstale":"2024-03-30T12:57:52.324312+0000","last_undegraded":"2024-03-30T12:57:52.324312+0000","last_fullsized":"2024-03-30T12:57:52.324312+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:56:59.916447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:59.916447+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:24.498514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":27,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-30T12:57:52.232891+0000","last_change":"2024-03-30T12:57:51.514427+0000","last_active":"2024-03-30T12:57:52.232891+0000","last_peered":"2024-03-30T12:57:52.232891+0000","last_clean":"2024-03-30T12:57:52.232891+0000","last_became_active":"2024-03-30T12:57:36.127063+0000","last_became_peered":"2024-03-30T12:57:36.127063+0000","last_unstale":"2024-03-30T12:57:52.232891+0000","last_undegraded":"2024-03-30T12:57:52.232891+0000","last_fullsized":"2024-03-30T12:57:52.232891+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T12:57:51.514144+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:54.967839+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":98,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-30T12:57:52.234411+0000","last_change":"2024-03-30T12:57:04.946767+0000","last_active":"2024-03-30T12:57:52.234411+0000","last_peered":"2024-03-30T12:57:52.234411+0000","last_clean":"2024-03-30T12:57:52.234411+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T12:57:52.234411+0000","last_undegraded":"2024-03-30T12:57:52.234411+0000","last_fullsized":"2024-03-30T12:57:52.234411+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:57:04.946489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:04.946489+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:26.951070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":112,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-30T12:57:52.324008+0000","last_change":"2024-03-30T12:56:48.287112+0000","last_active":"2024-03-30T12:57:52.324008+0000","last_peered":"2024-03-30T12:57:52.324008+0000","last_clean":"2024-03-30T12:57:52.324008+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T12:57:52.324008+0000","last_undegraded":"2024-03-30T12:57:52.324008+0000","last_fullsized":"2024-03-30T12:57:52.324008+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:56:48.286816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:48.286816+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:57:59.500682+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":34,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-30T12:57:52.233765+0000","last_change":"2024-03-30T12:57:36.264116+0000","last_active":"2024-03-30T12:57:52.233765+0000","last_peered":"2024-03-30T12:57:52.233765+0000","last_clean":"2024-03-30T12:57:52.233765+0000","last_became_active":"2024-03-30T12:57:36.129154+0000","last_became_peered":"2024-03-30T12:57:36.129154+0000","last_unstale":"2024-03-30T12:57:52.233765+0000","last_undegraded":"2024-03-30T12:57:52.233765+0000","last_fullsized":"2024-03-30T12:57:52.233765+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T12:57:02.954261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:02.954261+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:30.139627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":106,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-30T12:57:52.233223+0000","last_change":"2024-03-30T12:56:58.446137+0000","last_active":"2024-03-30T12:57:52.233223+0000","last_peered":"2024-03-30T12:57:52.233223+0000","last_clean":"2024-03-30T12:57:52.233223+0000","last_became_active":"2024-03-30T12:55:43.930722+0000","last_became_peered":"2024-03-30T12:55:43.930722+0000","last_unstale":"2024-03-30T12:57:52.233223+0000","last_undegraded":"2024-03-30T12:57:52.233223+0000","last_fullsized":"2024-03-30T12:57:52.233223+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:56:58.445737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:58.445737+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:21.311535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":112,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-30T12:57:52.324489+0000","last_change":"2024-03-30T12:57:36.254931+0000","last_active":"2024-03-30T12:57:52.324489+0000","last_peered":"2024-03-30T12:57:52.324489+0000","last_clean":"2024-03-30T12:57:52.324489+0000","last_became_active":"2024-03-30T12:57:36.140991+0000","last_became_peered":"2024-03-30T12:57:36.140991+0000","last_unstale":"2024-03-30T12:57:52.324489+0000","last_undegraded":"2024-03-30T12:57:52.324489+0000","last_fullsized":"2024-03-30T12:57:52.324489+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:00.268171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:00.268171+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:01.632312+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":70,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-30T12:57:52.233773+0000","last_change":"2024-03-30T12:57:16.932042+0000","last_active":"2024-03-30T12:57:52.233773+0000","last_peered":"2024-03-30T12:57:52.233773+0000","last_clean":"2024-03-30T12:57:52.233773+0000","last_became_active":"2024-03-30T12:57:16.930714+0000","last_became_peered":"2024-03-30T12:57:16.930714+0000","last_unstale":"2024-03-30T12:57:52.233773+0000","last_undegraded":"2024-03-30T12:57:52.233773+0000","last_fullsized":"2024-03-30T12:57:52.233773+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:57:11.300486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:11.300486+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:29.376314+0000","scrub_duration":503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":140,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-30T12:57:52.323460+0000","last_change":"2024-03-30T12:57:16.932035+0000","last_active":"2024-03-30T12:57:52.323460+0000","last_peered":"2024-03-30T12:57:52.323460+0000","last_clean":"2024-03-30T12:57:52.323460+0000","last_became_active":"2024-03-30T12:57:16.930513+0000","last_became_peered":"2024-03-30T12:57:16.930513+0000","last_unstale":"2024-03-30T12:57:52.323460+0000","last_undegraded":"2024-03-30T12:57:52.323460+0000","last_fullsized":"2024-03-30T12:57:52.323460+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:57:00.039850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:00.039850+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:15.642502+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":125,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-30T12:57:52.323604+0000","last_change":"2024-03-30T12:57:11.126991+0000","last_active":"2024-03-30T12:57:52.323604+0000","last_peered":"2024-03-30T12:57:52.323604+0000","last_clean":"2024-03-30T12:57:52.323604+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T12:57:52.323604+0000","last_undegraded":"2024-03-30T12:57:52.323604+0000","last_fullsized":"2024-03-30T12:57:52.323604+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T12:57:11.126678+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:20.211259+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":110,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-30T12:57:52.234465+0000","last_change":"2024-03-30T12:56:56.068730+0000","last_active":"2024-03-30T12:57:52.234465+0000","last_peered":"2024-03-30T12:57:52.234465+0000","last_clean":"2024-03-30T12:57:52.234465+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T12:57:52.234465+0000","last_undegraded":"2024-03-30T12:57:52.234465+0000","last_fullsized":"2024-03-30T12:57:52.234465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:56:56.068479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:56.068479+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:08.832176+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-30T12:57:52.234370+0000","last_change":"2024-03-30T12:57:36.165232+0000","last_active":"2024-03-30T12:57:52.234370+0000","last_peered":"2024-03-30T12:57:52.234370+0000","last_clean":"2024-03-30T12:57:52.234370+0000","last_became_active":"2024-03-30T12:57:36.129331+0000","last_became_peered":"2024-03-30T12:57:36.129331+0000","last_unstale":"2024-03-30T12:57:52.234370+0000","last_undegraded":"2024-03-30T12:57:52.234370+0000","last_fullsized":"2024-03-30T12:57:52.234370+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:56:56.870694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:56.870694+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:11.913038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":110,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-30T12:57:52.234651+0000","last_change":"2024-03-30T12:56:50.034844+0000","last_active":"2024-03-30T12:57:52.234651+0000","last_peered":"2024-03-30T12:57:52.234651+0000","last_clean":"2024-03-30T12:57:52.234651+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T12:57:52.234651+0000","last_undegraded":"2024-03-30T12:57:52.234651+0000","last_fullsized":"2024-03-30T12:57:52.234651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:56:50.034645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:50.034645+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:07.024201+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":101,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-03-30T12:57:52.232905+0000","last_change":"2024-03-30T12:57:09.558059+0000","last_active":"2024-03-30T12:57:52.232905+0000","last_peered":"2024-03-30T12:57:52.232905+0000","last_clean":"2024-03-30T12:57:52.232905+0000","last_became_active":"2024-03-30T12:55:43.925945+0000","last_became_peered":"2024-03-30T12:55:43.925945+0000","last_unstale":"2024-03-30T12:57:52.232905+0000","last_undegraded":"2024-03-30T12:57:52.232905+0000","last_fullsized":"2024-03-30T12:57:52.232905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:09.557910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:09.557910+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:37.248333+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":97,"num_read_kb":4618,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":59580023,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":104714240,"data_stored":104714240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509471,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42752,"kb_used_data":15732,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329088,"statfs":{"total":96636764160,"available":96592986112,"internally_reserved":0,"allocated":16109568,"data_stored":15734471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542176,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46916,"kb_used_data":19896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94324924,"statfs":{"total":96636764160,"available":96588722176,"internally_reserved":0,"allocated":20373504,"data_stored":19998407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822665,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39712,"kb_used_data":12180,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94332128,"statfs":{"total":96636764160,"available":96596099072,"internally_reserved":0,"allocated":12472320,"data_stored":12097223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4399,"internal_metadata":28176081},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509472,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40064,"kb_used_data":13044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331776,"statfs":{"total":96636764160,"available":96595738624,"internally_reserved":0,"allocated":13357056,"data_stored":12981959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509472,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36120,"kb_used_data":9100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94335720,"statfs":{"total":96636764160,"available":96599777280,"internally_reserved":0,"allocated":9318400,"data_stored":8943303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509472,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41144,"kb_used_data":14124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330696,"statfs":{"total":96636764160,"available":96594632704,"internally_reserved":0,"allocated":14462976,"data_stored":14087879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574881,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40144,"kb_used_data":13124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331696,"statfs":{"total":96636764160,"available":96595656704,"internally_reserved":0,"allocated":13438976,"data_stored":13063879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8679424,"data_stored":8679424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12718080,"data_stored":12718080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19734528,"data_stored":19734528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9654272,"data_stored":9654272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:58:00.786 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T12:58:00.786 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T12:58:00.786 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2024-03-30T12:58:00.786 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-30T12:58:00.787 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-30T12:58:00.789 INFO:tasks.ceph.osd.7:Started 2024-03-30T12:58:00.789 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-30T12:58:00.859 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:58:00.855+0000 7fb1f2f8e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:00.867 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:58:00.863+0000 7fb1f2f8e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:00.867 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:58:00.863+0000 7fb1f2f8e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:00.906 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T12:58:00.907 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T12:58:00.907 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T12:58:00.907 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T12:58:00.907 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T12:58:00.907 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T12:58:00.907 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-03-30T12:58:00.907 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-03-30T12:58:00.907 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-03-30T12:58:00.907 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-03-30T12:58:00.907 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-03-30T12:58:00.907 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T12:58:00.909 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T12:58:00.909 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-03-30T12:58:01.413 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:58:01.407+0000 7fb1f2f8e7c0 -1 Falling back to public interface 2024-03-30T12:58:05.103 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:58:05.099+0000 7fb1f2f8e7c0 -1 osd.7 52 log_to_monitors true 2024-03-30T12:58:05.315 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:58:05.311+0000 7fb1ead63640 -1 osd.7 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T12:58:05.910 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-30T12:58:06.043 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T12:58:06.043 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T12:58:06.043 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T12:58:06.043 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T12:58:06.057 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T12:58:06.057 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-03-30T12:58:06.057 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 7 2024-03-30T12:58:06.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:06.180+0000 7f9dc28c7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:06.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:06.196+0000 7f9dc28c7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:06.385 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T12:58:06.380+0000 7fbb38f08640 -1 mon.a@0(leader).osd e59 definitely_dead 0 2024-03-30T12:58:07.291 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.7. 2024-03-30T12:58:07.321 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-03-30T12:58:07.322 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T12:58:07.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:07.432+0000 7fdd3e513640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:07.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:07.448+0000 7fdd3e513640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:08.307 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T12:58:08.334 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T12:58:08.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:08.444+0000 7f072b94c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:08.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:08.460+0000 7f072b94c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:09.296 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T12:58:09.332 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T12:58:09.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:09.448+0000 7f0e6997f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:09.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:09.464+0000 7f0e6997f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:10.324 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T12:58:10.352 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T12:58:10.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:10.464+0000 7f8e42595640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:10.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:10.480+0000 7f8e42595640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:11.345 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T12:58:11.374 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T12:58:11.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:11.492+0000 7f25c052e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:11.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:11.508+0000 7f25c052e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:12.355 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T12:58:12.382 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T12:58:12.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:12.496+0000 7f5b4314c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:12.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:12.512+0000 7f5b4314c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:13.357 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T12:58:13.391 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T12:58:13.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:13.508+0000 7fc93b2dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:13.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:13.528+0000 7fc93b2dc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:14.380 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T12:58:14.405 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T12:58:14.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:14.520+0000 7ff475635640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:14.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:14.536+0000 7ff475635640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:15.384 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T12:58:15.412 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T12:58:15.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:15.528+0000 7f441a0aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:15.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:15.544+0000 7f441a0aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:16.402 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T12:58:16.428 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T12:58:16.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:16.548+0000 7f7ea624c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:16.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:16.564+0000 7f7ea624c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:17.415 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T12:58:17.439 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T12:58:17.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:17.552+0000 7feb9d52b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:17.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:17.568+0000 7feb9d52b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:18.431 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T12:58:18.457 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T12:58:18.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:18.576+0000 7f789ff82640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:18.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:18.596+0000 7f789ff82640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:19.428 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T12:58:19.454 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T12:58:19.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:19.572+0000 7f58fa4aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:19.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:19.588+0000 7f58fa4aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:20.446 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T12:58:20.475 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T12:58:20.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:20.588+0000 7f01f9474640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:20.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:20.604+0000 7f01f9474640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:21.455 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T12:58:21.485 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T12:58:21.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:21.600+0000 7f89e4da5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:21.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:21.620+0000 7f89e4da5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:22.476 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T12:58:22.501 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T12:58:22.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:22.616+0000 7f125ba69640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:22.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:22.632+0000 7f125ba69640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:23.486 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T12:58:23.512 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T12:58:23.513 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:58:23.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:23.628+0000 7fc5b361e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:23.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:23.640+0000 7fc5b361e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:23.821 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:58:23.822 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T12:58:23.840 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-03-30T12:58:23.487848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":97,"num_read_kb":4618,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44621991,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":317232,"kb_used_data":100016,"kb_used_omap":17,"kb_used_meta":217070,"kb_avail":754657488,"statfs":{"total":773094113280,"available":772769267712,"internally_reserved":0,"allocated":102416384,"data_stored":98601176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18361,"internal_metadata":222279751},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.055572"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":148,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-30T12:58:18.900995+0000","last_change":"2024-03-30T12:57:16.932276+0000","last_active":"2024-03-30T12:58:18.900995+0000","last_peered":"2024-03-30T12:58:18.900995+0000","last_clean":"2024-03-30T12:58:18.900995+0000","last_became_active":"2024-03-30T12:57:16.931067+0000","last_became_peered":"2024-03-30T12:57:16.931067+0000","last_unstale":"2024-03-30T12:58:18.900995+0000","last_undegraded":"2024-03-30T12:58:18.900995+0000","last_fullsized":"2024-03-30T12:58:18.900995+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:12.143245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:12.143245+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:38.357453+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":118,"reported_epoch":72,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-30T12:58:19.435091+0000","last_change":"2024-03-30T12:58:09.318656+0000","last_active":"2024-03-30T12:58:19.435091+0000","last_peered":"2024-03-30T12:58:19.435091+0000","last_clean":"2024-03-30T12:58:06.286271+0000","last_became_active":"2024-03-30T12:58:09.314481+0000","last_became_peered":"2024-03-30T12:58:09.314481+0000","last_unstale":"2024-03-30T12:58:19.435091+0000","last_undegraded":"2024-03-30T12:58:19.435091+0000","last_fullsized":"2024-03-30T12:58:19.435091+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:56:54.949876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:54.949876+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,1,3],"avail_no_missing":["4(0)","1(1)","3(2)","7(1)"],"object_location_counts":[{"shards":"1(1),3(2),4(0)","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":72,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-30T12:58:20.911680+0000","last_change":"2024-03-30T12:58:11.362467+0000","last_active":"2024-03-30T12:58:20.911680+0000","last_peered":"2024-03-30T12:58:20.911680+0000","last_clean":"2024-03-30T12:58:20.911680+0000","last_became_active":"2024-03-30T12:58:11.361337+0000","last_became_peered":"2024-03-30T12:58:11.361337+0000","last_unstale":"2024-03-30T12:58:20.911680+0000","last_undegraded":"2024-03-30T12:58:20.911680+0000","last_fullsized":"2024-03-30T12:58:20.911680+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:56:59.916447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:59.916447+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:21.863264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":70,"reported_epoch":72,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-30T12:58:19.433008+0000","last_change":"2024-03-30T12:58:09.324631+0000","last_active":"2024-03-30T12:58:19.433008+0000","last_peered":"2024-03-30T12:58:19.433008+0000","last_clean":"2024-03-30T12:58:06.282146+0000","last_became_active":"2024-03-30T12:58:09.313288+0000","last_became_peered":"2024-03-30T12:58:09.313288+0000","last_unstale":"2024-03-30T12:58:19.433008+0000","last_undegraded":"2024-03-30T12:58:19.433008+0000","last_fullsized":"2024-03-30T12:58:19.433008+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T12:57:51.514144+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7008420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[1,0,6],"avail_no_missing":["1(0)","0(1)","6(2)","7(0)"],"object_location_counts":[{"shards":"0(1),1(0),6(2)","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":127,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-30T12:58:18.900855+0000","last_change":"2024-03-30T12:57:04.946767+0000","last_active":"2024-03-30T12:58:18.900855+0000","last_peered":"2024-03-30T12:58:18.900855+0000","last_clean":"2024-03-30T12:58:18.900855+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T12:58:18.900855+0000","last_undegraded":"2024-03-30T12:58:18.900855+0000","last_fullsized":"2024-03-30T12:58:18.900855+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:57:04.946489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:04.946489+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:26.951070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":155,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-30T12:58:21.880833+0000","last_change":"2024-03-30T12:58:01.131164+0000","last_active":"2024-03-30T12:58:21.880833+0000","last_peered":"2024-03-30T12:58:21.880833+0000","last_clean":"2024-03-30T12:58:21.880833+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T12:58:21.880833+0000","last_undegraded":"2024-03-30T12:58:21.880833+0000","last_fullsized":"2024-03-30T12:58:21.880833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:58:01.130774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:01.130774+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:29.177753+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":114,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-30T12:58:20.911518+0000","last_change":"2024-03-30T12:58:13.024975+0000","last_active":"2024-03-30T12:58:20.911518+0000","last_peered":"2024-03-30T12:58:20.911518+0000","last_clean":"2024-03-30T12:58:20.911518+0000","last_became_active":"2024-03-30T12:58:12.369218+0000","last_became_peered":"2024-03-30T12:58:12.369218+0000","last_unstale":"2024-03-30T12:58:20.911518+0000","last_undegraded":"2024-03-30T12:58:20.911518+0000","last_fullsized":"2024-03-30T12:58:20.911518+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T12:58:13.024706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:13.024706+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:36.211866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":138,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T12:58:19.433191+0000","last_change":"2024-03-30T12:56:58.446137+0000","last_active":"2024-03-30T12:58:19.433191+0000","last_peered":"2024-03-30T12:58:19.433191+0000","last_clean":"2024-03-30T12:58:19.433191+0000","last_became_active":"2024-03-30T12:55:43.930722+0000","last_became_peered":"2024-03-30T12:55:43.930722+0000","last_unstale":"2024-03-30T12:58:19.433191+0000","last_undegraded":"2024-03-30T12:58:19.433191+0000","last_fullsized":"2024-03-30T12:58:19.433191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:56:58.445737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:58.445737+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:21.311535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":173,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-30T12:58:21.881028+0000","last_change":"2024-03-30T12:58:11.358195+0000","last_active":"2024-03-30T12:58:21.881028+0000","last_peered":"2024-03-30T12:58:21.881028+0000","last_clean":"2024-03-30T12:58:21.881028+0000","last_became_active":"2024-03-30T12:58:11.357621+0000","last_became_peered":"2024-03-30T12:58:11.357621+0000","last_unstale":"2024-03-30T12:58:21.881028+0000","last_undegraded":"2024-03-30T12:58:21.881028+0000","last_fullsized":"2024-03-30T12:58:21.881028+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:58:04.171994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:04.171994+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:23.627732+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":102,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T12:58:19.434961+0000","last_change":"2024-03-30T12:57:16.932042+0000","last_active":"2024-03-30T12:58:19.434961+0000","last_peered":"2024-03-30T12:58:19.434961+0000","last_clean":"2024-03-30T12:58:19.434961+0000","last_became_active":"2024-03-30T12:57:16.930714+0000","last_became_peered":"2024-03-30T12:57:16.930714+0000","last_unstale":"2024-03-30T12:58:19.434961+0000","last_undegraded":"2024-03-30T12:58:19.434961+0000","last_fullsized":"2024-03-30T12:58:19.434961+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:57:11.300486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:11.300486+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:29.376314+0000","scrub_duration":503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":177,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-30T12:58:19.389618+0000","last_change":"2024-03-30T12:58:19.389708+0000","last_active":"2024-03-30T12:58:19.389618+0000","last_peered":"2024-03-30T12:58:19.389618+0000","last_clean":"2024-03-30T12:58:19.389618+0000","last_became_active":"2024-03-30T12:57:16.930513+0000","last_became_peered":"2024-03-30T12:57:16.930513+0000","last_unstale":"2024-03-30T12:58:19.389618+0000","last_undegraded":"2024-03-30T12:58:19.389618+0000","last_fullsized":"2024-03-30T12:58:19.389618+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:58:19.389431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:19.389431+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:31.361294+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":154,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-30T12:58:18.901237+0000","last_change":"2024-03-30T12:57:11.126991+0000","last_active":"2024-03-30T12:58:18.901237+0000","last_peered":"2024-03-30T12:58:18.901237+0000","last_clean":"2024-03-30T12:58:18.901237+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T12:58:18.901237+0000","last_undegraded":"2024-03-30T12:58:18.901237+0000","last_fullsized":"2024-03-30T12:58:18.901237+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T12:57:11.126678+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:20.211259+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":147,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-30T12:58:18.900717+0000","last_change":"2024-03-30T12:58:14.711329+0000","last_active":"2024-03-30T12:58:18.900717+0000","last_peered":"2024-03-30T12:58:18.900717+0000","last_clean":"2024-03-30T12:58:18.900717+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T12:58:18.900717+0000","last_undegraded":"2024-03-30T12:58:18.900717+0000","last_fullsized":"2024-03-30T12:58:18.900717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:58:14.710869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:14.710869+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:25.694672+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":52,"reported_epoch":71,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-30T12:58:18.900770+0000","last_change":"2024-03-30T12:58:09.319904+0000","last_active":"2024-03-30T12:58:18.900770+0000","last_peered":"2024-03-30T12:58:18.900770+0000","last_clean":"2024-03-30T12:58:06.285343+0000","last_became_active":"2024-03-30T12:58:09.316916+0000","last_became_peered":"2024-03-30T12:58:09.316916+0000","last_unstale":"2024-03-30T12:58:18.900770+0000","last_undegraded":"2024-03-30T12:58:18.900770+0000","last_fullsized":"2024-03-30T12:58:18.900770+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:56:56.870694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:56.870694+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[5,3,2],"avail_no_missing":["5(0)","2(2)","3(1)","7(0)"],"object_location_counts":[{"shards":"2(2),3(1),5(0)","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":147,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-30T12:58:18.900757+0000","last_change":"2024-03-30T12:58:07.734464+0000","last_active":"2024-03-30T12:58:18.900757+0000","last_peered":"2024-03-30T12:58:18.900757+0000","last_clean":"2024-03-30T12:58:18.900757+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T12:58:18.900757+0000","last_undegraded":"2024-03-30T12:58:18.900757+0000","last_fullsized":"2024-03-30T12:58:18.900757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:58:07.734198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:07.734198+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:13.487012+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":134,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T12:58:19.433006+0000","last_change":"2024-03-30T12:57:09.558059+0000","last_active":"2024-03-30T12:58:19.433006+0000","last_peered":"2024-03-30T12:58:19.433006+0000","last_clean":"2024-03-30T12:58:19.433006+0000","last_became_active":"2024-03-30T12:55:43.925945+0000","last_became_peered":"2024-03-30T12:55:43.925945+0000","last_unstale":"2024-03-30T12:58:19.433006+0000","last_undegraded":"2024-03-30T12:58:19.433006+0000","last_fullsized":"2024-03-30T12:58:19.433006+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:09.557910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:09.557910+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:37.248333+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":97,"num_read_kb":4618,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44621991,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":95780864,"data_stored":95780864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509476,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42936,"kb_used_data":15912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94328904,"statfs":{"total":96636764160,"available":96592797696,"internally_reserved":0,"allocated":16293888,"data_stored":15820338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542180,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47088,"kb_used_data":20064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94324752,"statfs":{"total":96636764160,"available":96588546048,"internally_reserved":0,"allocated":20545536,"data_stored":20078688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822670,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39912,"kb_used_data":12372,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94331928,"statfs":{"total":96636764160,"available":96595894272,"internally_reserved":0,"allocated":12668928,"data_stored":12188676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4399,"internal_metadata":28176081},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509477,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34148,"kb_used_data":7128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337692,"statfs":{"total":96636764160,"available":96601796608,"internally_reserved":0,"allocated":7299072,"data_stored":6825522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509477,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34976,"kb_used_data":7948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336864,"statfs":{"total":96636764160,"available":96600948736,"internally_reserved":0,"allocated":8138752,"data_stored":7665202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509477,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41320,"kb_used_data":14292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330520,"statfs":{"total":96636764160,"available":96594452480,"internally_reserved":0,"allocated":14635008,"data_stored":14168160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":59,"seq":253403070469,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36492,"kb_used_data":8960,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94335348,"statfs":{"total":96636764160,"available":96599396352,"internally_reserved":0,"allocated":9175040,"data_stored":8688086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4391,"internal_metadata":28176089},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574886,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40360,"kb_used_data":13340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331480,"statfs":{"total":96636764160,"available":96595435520,"internally_reserved":0,"allocated":13660160,"data_stored":13166504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7315456,"data_stored":7315456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19734528,"data_stored":19734528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8327168,"data_stored":8327168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:58:23.841 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:58:23.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:23.948+0000 7fcc1b751640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:23.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:23.960+0000 7fcc1b751640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:24.137 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:58:24.138 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T12:58:24.155 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2024-03-30T12:58:23.829064+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":97,"num_read_kb":4618,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44621991,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":317232,"kb_used_data":100016,"kb_used_omap":17,"kb_used_meta":217070,"kb_avail":754657488,"statfs":{"total":773094113280,"available":772769267712,"internally_reserved":0,"allocated":102416384,"data_stored":98601176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18361,"internal_metadata":222279751},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001169"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":148,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-30T12:58:18.900995+0000","last_change":"2024-03-30T12:57:16.932276+0000","last_active":"2024-03-30T12:58:18.900995+0000","last_peered":"2024-03-30T12:58:18.900995+0000","last_clean":"2024-03-30T12:58:18.900995+0000","last_became_active":"2024-03-30T12:57:16.931067+0000","last_became_peered":"2024-03-30T12:57:16.931067+0000","last_unstale":"2024-03-30T12:58:18.900995+0000","last_undegraded":"2024-03-30T12:58:18.900995+0000","last_fullsized":"2024-03-30T12:58:18.900995+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:12.143245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:12.143245+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:38.357453+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":118,"reported_epoch":72,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-30T12:58:19.435091+0000","last_change":"2024-03-30T12:58:09.318656+0000","last_active":"2024-03-30T12:58:19.435091+0000","last_peered":"2024-03-30T12:58:19.435091+0000","last_clean":"2024-03-30T12:58:06.286271+0000","last_became_active":"2024-03-30T12:58:09.314481+0000","last_became_peered":"2024-03-30T12:58:09.314481+0000","last_unstale":"2024-03-30T12:58:19.435091+0000","last_undegraded":"2024-03-30T12:58:19.435091+0000","last_fullsized":"2024-03-30T12:58:19.435091+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:56:54.949876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:54.949876+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,1,3],"avail_no_missing":["4(0)","1(1)","3(2)","7(1)"],"object_location_counts":[{"shards":"1(1),3(2),4(0)","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":72,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-30T12:58:20.911680+0000","last_change":"2024-03-30T12:58:11.362467+0000","last_active":"2024-03-30T12:58:20.911680+0000","last_peered":"2024-03-30T12:58:20.911680+0000","last_clean":"2024-03-30T12:58:20.911680+0000","last_became_active":"2024-03-30T12:58:11.361337+0000","last_became_peered":"2024-03-30T12:58:11.361337+0000","last_unstale":"2024-03-30T12:58:20.911680+0000","last_undegraded":"2024-03-30T12:58:20.911680+0000","last_fullsized":"2024-03-30T12:58:20.911680+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:56:59.916447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:59.916447+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:21.863264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":70,"reported_epoch":72,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-30T12:58:19.433008+0000","last_change":"2024-03-30T12:58:09.324631+0000","last_active":"2024-03-30T12:58:19.433008+0000","last_peered":"2024-03-30T12:58:19.433008+0000","last_clean":"2024-03-30T12:58:06.282146+0000","last_became_active":"2024-03-30T12:58:09.313288+0000","last_became_peered":"2024-03-30T12:58:09.313288+0000","last_unstale":"2024-03-30T12:58:19.433008+0000","last_undegraded":"2024-03-30T12:58:19.433008+0000","last_fullsized":"2024-03-30T12:58:19.433008+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T12:57:51.514144+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7008420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[1,0,6],"avail_no_missing":["1(0)","0(1)","6(2)","7(0)"],"object_location_counts":[{"shards":"0(1),1(0),6(2)","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":127,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-30T12:58:18.900855+0000","last_change":"2024-03-30T12:57:04.946767+0000","last_active":"2024-03-30T12:58:18.900855+0000","last_peered":"2024-03-30T12:58:18.900855+0000","last_clean":"2024-03-30T12:58:18.900855+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T12:58:18.900855+0000","last_undegraded":"2024-03-30T12:58:18.900855+0000","last_fullsized":"2024-03-30T12:58:18.900855+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:57:04.946489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:04.946489+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:26.951070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":155,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-30T12:58:21.880833+0000","last_change":"2024-03-30T12:58:01.131164+0000","last_active":"2024-03-30T12:58:21.880833+0000","last_peered":"2024-03-30T12:58:21.880833+0000","last_clean":"2024-03-30T12:58:21.880833+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T12:58:21.880833+0000","last_undegraded":"2024-03-30T12:58:21.880833+0000","last_fullsized":"2024-03-30T12:58:21.880833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:58:01.130774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:01.130774+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:29.177753+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":114,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-30T12:58:20.911518+0000","last_change":"2024-03-30T12:58:13.024975+0000","last_active":"2024-03-30T12:58:20.911518+0000","last_peered":"2024-03-30T12:58:20.911518+0000","last_clean":"2024-03-30T12:58:20.911518+0000","last_became_active":"2024-03-30T12:58:12.369218+0000","last_became_peered":"2024-03-30T12:58:12.369218+0000","last_unstale":"2024-03-30T12:58:20.911518+0000","last_undegraded":"2024-03-30T12:58:20.911518+0000","last_fullsized":"2024-03-30T12:58:20.911518+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T12:58:13.024706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:13.024706+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:36.211866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":138,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T12:58:19.433191+0000","last_change":"2024-03-30T12:56:58.446137+0000","last_active":"2024-03-30T12:58:19.433191+0000","last_peered":"2024-03-30T12:58:19.433191+0000","last_clean":"2024-03-30T12:58:19.433191+0000","last_became_active":"2024-03-30T12:55:43.930722+0000","last_became_peered":"2024-03-30T12:55:43.930722+0000","last_unstale":"2024-03-30T12:58:19.433191+0000","last_undegraded":"2024-03-30T12:58:19.433191+0000","last_fullsized":"2024-03-30T12:58:19.433191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:56:58.445737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:58.445737+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:21.311535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":173,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-30T12:58:21.881028+0000","last_change":"2024-03-30T12:58:11.358195+0000","last_active":"2024-03-30T12:58:21.881028+0000","last_peered":"2024-03-30T12:58:21.881028+0000","last_clean":"2024-03-30T12:58:21.881028+0000","last_became_active":"2024-03-30T12:58:11.357621+0000","last_became_peered":"2024-03-30T12:58:11.357621+0000","last_unstale":"2024-03-30T12:58:21.881028+0000","last_undegraded":"2024-03-30T12:58:21.881028+0000","last_fullsized":"2024-03-30T12:58:21.881028+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:58:04.171994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:04.171994+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:23.627732+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":102,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T12:58:19.434961+0000","last_change":"2024-03-30T12:57:16.932042+0000","last_active":"2024-03-30T12:58:19.434961+0000","last_peered":"2024-03-30T12:58:19.434961+0000","last_clean":"2024-03-30T12:58:19.434961+0000","last_became_active":"2024-03-30T12:57:16.930714+0000","last_became_peered":"2024-03-30T12:57:16.930714+0000","last_unstale":"2024-03-30T12:58:19.434961+0000","last_undegraded":"2024-03-30T12:58:19.434961+0000","last_fullsized":"2024-03-30T12:58:19.434961+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:57:11.300486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:11.300486+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:29.376314+0000","scrub_duration":503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":177,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-30T12:58:19.389618+0000","last_change":"2024-03-30T12:58:19.389708+0000","last_active":"2024-03-30T12:58:19.389618+0000","last_peered":"2024-03-30T12:58:19.389618+0000","last_clean":"2024-03-30T12:58:19.389618+0000","last_became_active":"2024-03-30T12:57:16.930513+0000","last_became_peered":"2024-03-30T12:57:16.930513+0000","last_unstale":"2024-03-30T12:58:19.389618+0000","last_undegraded":"2024-03-30T12:58:19.389618+0000","last_fullsized":"2024-03-30T12:58:19.389618+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:58:19.389431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:19.389431+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:31.361294+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":154,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-30T12:58:18.901237+0000","last_change":"2024-03-30T12:57:11.126991+0000","last_active":"2024-03-30T12:58:18.901237+0000","last_peered":"2024-03-30T12:58:18.901237+0000","last_clean":"2024-03-30T12:58:18.901237+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T12:58:18.901237+0000","last_undegraded":"2024-03-30T12:58:18.901237+0000","last_fullsized":"2024-03-30T12:58:18.901237+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T12:57:11.126678+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:20.211259+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":147,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-30T12:58:18.900717+0000","last_change":"2024-03-30T12:58:14.711329+0000","last_active":"2024-03-30T12:58:18.900717+0000","last_peered":"2024-03-30T12:58:18.900717+0000","last_clean":"2024-03-30T12:58:18.900717+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T12:58:18.900717+0000","last_undegraded":"2024-03-30T12:58:18.900717+0000","last_fullsized":"2024-03-30T12:58:18.900717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:58:14.710869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:14.710869+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:25.694672+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":52,"reported_epoch":71,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-30T12:58:18.900770+0000","last_change":"2024-03-30T12:58:09.319904+0000","last_active":"2024-03-30T12:58:18.900770+0000","last_peered":"2024-03-30T12:58:18.900770+0000","last_clean":"2024-03-30T12:58:06.285343+0000","last_became_active":"2024-03-30T12:58:09.316916+0000","last_became_peered":"2024-03-30T12:58:09.316916+0000","last_unstale":"2024-03-30T12:58:18.900770+0000","last_undegraded":"2024-03-30T12:58:18.900770+0000","last_fullsized":"2024-03-30T12:58:18.900770+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:56:56.870694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:56.870694+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[5,3,2],"avail_no_missing":["5(0)","2(2)","3(1)","7(0)"],"object_location_counts":[{"shards":"2(2),3(1),5(0)","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":147,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-30T12:58:18.900757+0000","last_change":"2024-03-30T12:58:07.734464+0000","last_active":"2024-03-30T12:58:18.900757+0000","last_peered":"2024-03-30T12:58:18.900757+0000","last_clean":"2024-03-30T12:58:18.900757+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T12:58:18.900757+0000","last_undegraded":"2024-03-30T12:58:18.900757+0000","last_fullsized":"2024-03-30T12:58:18.900757+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:58:07.734198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:07.734198+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:13.487012+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":134,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T12:58:19.433006+0000","last_change":"2024-03-30T12:57:09.558059+0000","last_active":"2024-03-30T12:58:19.433006+0000","last_peered":"2024-03-30T12:58:19.433006+0000","last_clean":"2024-03-30T12:58:19.433006+0000","last_became_active":"2024-03-30T12:55:43.925945+0000","last_became_peered":"2024-03-30T12:55:43.925945+0000","last_unstale":"2024-03-30T12:58:19.433006+0000","last_undegraded":"2024-03-30T12:58:19.433006+0000","last_fullsized":"2024-03-30T12:58:19.433006+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:09.557910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:09.557910+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:37.248333+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":97,"num_read_kb":4618,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44621991,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":95780864,"data_stored":95780864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509476,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42936,"kb_used_data":15912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94328904,"statfs":{"total":96636764160,"available":96592797696,"internally_reserved":0,"allocated":16293888,"data_stored":15820338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542180,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47088,"kb_used_data":20064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94324752,"statfs":{"total":96636764160,"available":96588546048,"internally_reserved":0,"allocated":20545536,"data_stored":20078688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822670,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39912,"kb_used_data":12372,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94331928,"statfs":{"total":96636764160,"available":96595894272,"internally_reserved":0,"allocated":12668928,"data_stored":12188676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4399,"internal_metadata":28176081},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509477,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34148,"kb_used_data":7128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337692,"statfs":{"total":96636764160,"available":96601796608,"internally_reserved":0,"allocated":7299072,"data_stored":6825522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509477,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34976,"kb_used_data":7948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336864,"statfs":{"total":96636764160,"available":96600948736,"internally_reserved":0,"allocated":8138752,"data_stored":7665202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509477,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41320,"kb_used_data":14292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330520,"statfs":{"total":96636764160,"available":96594452480,"internally_reserved":0,"allocated":14635008,"data_stored":14168160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":59,"seq":253403070469,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36492,"kb_used_data":8960,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94335348,"statfs":{"total":96636764160,"available":96599396352,"internally_reserved":0,"allocated":9175040,"data_stored":8688086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4391,"internal_metadata":28176089},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574886,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40360,"kb_used_data":13340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331480,"statfs":{"total":96636764160,"available":96595435520,"internally_reserved":0,"allocated":13660160,"data_stored":13166504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7315456,"data_stored":7315456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19734528,"data_stored":19734528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8327168,"data_stored":8327168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:58:24.156 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T12:58:24.157 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T12:58:24.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:24.264+0000 7f874d8dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:24.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:24.276+0000 7f874d8dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:24.457 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:58:24.457 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":76,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T12:58:23.476365+0000","last_up_change":"2024-03-30T12:58:06.276160+0000","last_in_change":"2024-03-30T12:58:07.288404+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3146012944},{"type":"v1","addr":"172.21.15.67:6818","nonce":3146012944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3146012944},{"type":"v1","addr":"172.21.15.67:6821","nonce":3146012944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3146012944},{"type":"v1","addr":"172.21.15.67:6828","nonce":3146012944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":3146012944},{"type":"v1","addr":"172.21.15.67:6825","nonce":3146012944}]},"public_addr":"172.21.15.67:6818/3146012944","cluster_addr":"172.21.15.67:6821/3146012944","heartbeat_back_addr":"172.21.15.67:6828/3146012944","heartbeat_front_addr":"172.21.15.67:6825/3146012944","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6801","nonce":1504348724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6803","nonce":1504348724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6807","nonce":1504348724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6805","nonce":1504348724}]},"public_addr":"172.21.15.67:6801/1504348724","cluster_addr":"172.21.15.67:6803/1504348724","heartbeat_back_addr":"172.21.15.67:6807/1504348724","heartbeat_front_addr":"172.21.15.67:6805/1504348724","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":25,"up_from":36,"up_thru":63,"down_at":26,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1927174653},{"type":"v1","addr":"172.21.15.82:6803","nonce":1927174653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":1927174653},{"type":"v1","addr":"172.21.15.82:6807","nonce":1927174653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":1927174653},{"type":"v1","addr":"172.21.15.82:6815","nonce":1927174653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":1927174653},{"type":"v1","addr":"172.21.15.82:6811","nonce":1927174653}]},"public_addr":"172.21.15.82:6803/1927174653","cluster_addr":"172.21.15.82:6807/1927174653","heartbeat_back_addr":"172.21.15.82:6815/1927174653","heartbeat_front_addr":"172.21.15.82:6811/1927174653","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6817","nonce":3467856022}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6819","nonce":3467856022}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6823","nonce":3467856022}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6821","nonce":3467856022}]},"public_addr":"172.21.15.82:6817/3467856022","cluster_addr":"172.21.15.82:6819/3467856022","heartbeat_back_addr":"172.21.15.82:6823/3467856022","heartbeat_front_addr":"172.21.15.82:6821/3467856022","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6801","nonce":968508548}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6806","nonce":968508548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6814","nonce":968508548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6810","nonce":968508548}]},"public_addr":"172.21.15.82:6801/968508548","cluster_addr":"172.21.15.82:6806/968508548","heartbeat_back_addr":"172.21.15.82:6814/968508548","heartbeat_front_addr":"172.21.15.82:6810/968508548","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":52,"up_from":59,"up_thru":64,"down_at":53,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":1490604388},{"type":"v1","addr":"172.21.15.82:6825","nonce":1490604388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":1490604388},{"type":"v1","addr":"172.21.15.82:6827","nonce":1490604388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":1490604388},{"type":"v1","addr":"172.21.15.82:6831","nonce":1490604388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":1490604388},{"type":"v1","addr":"172.21.15.82:6829","nonce":1490604388}]},"public_addr":"172.21.15.82:6825/1490604388","cluster_addr":"172.21.15.82:6827/1490604388","heartbeat_back_addr":"172.21.15.82:6831/1490604388","heartbeat_front_addr":"172.21.15.82:6829/1490604388","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:57:51.786809+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:58:01.969820+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:58:01.545062+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:58:13.383508+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:57:15.131405+0000","dead_epoch":26},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:58:11.960109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:58:10.360171+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:58:06.145865+0000","dead_epoch":53}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"1.0","osds":[1,0,6]},{"pgid":"1.9","osds":[5,3,2]},{"pgid":"1.e","osds":[4,1,3]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T12:58:24.475 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T12:58:24.475 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T12:58:24.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:24.576+0000 7faaa2e58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:24.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:24.592+0000 7faaa2e58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:24.763 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T12:58:24.780 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T12:58:24.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:24.904+0000 7f356cfad640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:24.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:24.924+0000 7f356cfad640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:25.106 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:58:25.106 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T12:58:25.126 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T12:58:25.126 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T12:58:25.126 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T12:58:25.126 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T12:58:25.127 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 1, 2, 3, 5, 6, 4, 7] 2024-03-30T12:58:25.127 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T12:58:25.127 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:58:25.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:25.232+0000 7ff915b50640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:25.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:25.244+0000 7ff915b50640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:25.415 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:58:25.416 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T12:58:25.432 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-03-30T12:58:24.494416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":97,"num_read_kb":4618,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44621991,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":51,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":317304,"kb_used_data":100088,"kb_used_omap":17,"kb_used_meta":217070,"kb_avail":754657416,"statfs":{"total":773094113280,"available":772769193984,"internally_reserved":0,"allocated":102490112,"data_stored":98634692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18361,"internal_metadata":222279751},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.050648"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":148,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-30T12:58:18.900995+0000","last_change":"2024-03-30T12:57:16.932276+0000","last_active":"2024-03-30T12:58:18.900995+0000","last_peered":"2024-03-30T12:58:18.900995+0000","last_clean":"2024-03-30T12:58:18.900995+0000","last_became_active":"2024-03-30T12:57:16.931067+0000","last_became_peered":"2024-03-30T12:57:16.931067+0000","last_unstale":"2024-03-30T12:58:18.900995+0000","last_undegraded":"2024-03-30T12:58:18.900995+0000","last_fullsized":"2024-03-30T12:58:18.900995+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:12.143245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:12.143245+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:38.357453+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":118,"reported_epoch":72,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-30T12:58:19.435091+0000","last_change":"2024-03-30T12:58:09.318656+0000","last_active":"2024-03-30T12:58:19.435091+0000","last_peered":"2024-03-30T12:58:19.435091+0000","last_clean":"2024-03-30T12:58:06.286271+0000","last_became_active":"2024-03-30T12:58:09.314481+0000","last_became_peered":"2024-03-30T12:58:09.314481+0000","last_unstale":"2024-03-30T12:58:19.435091+0000","last_undegraded":"2024-03-30T12:58:19.435091+0000","last_fullsized":"2024-03-30T12:58:19.435091+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:56:54.949876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:54.949876+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,1,3],"avail_no_missing":["4(0)","1(1)","3(2)","7(1)"],"object_location_counts":[{"shards":"1(1),3(2),4(0)","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":72,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-30T12:58:20.911680+0000","last_change":"2024-03-30T12:58:11.362467+0000","last_active":"2024-03-30T12:58:20.911680+0000","last_peered":"2024-03-30T12:58:20.911680+0000","last_clean":"2024-03-30T12:58:20.911680+0000","last_became_active":"2024-03-30T12:58:11.361337+0000","last_became_peered":"2024-03-30T12:58:11.361337+0000","last_unstale":"2024-03-30T12:58:20.911680+0000","last_undegraded":"2024-03-30T12:58:20.911680+0000","last_fullsized":"2024-03-30T12:58:20.911680+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:56:59.916447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:59.916447+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:21.863264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":70,"reported_epoch":72,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-30T12:58:19.433008+0000","last_change":"2024-03-30T12:58:09.324631+0000","last_active":"2024-03-30T12:58:19.433008+0000","last_peered":"2024-03-30T12:58:19.433008+0000","last_clean":"2024-03-30T12:58:06.282146+0000","last_became_active":"2024-03-30T12:58:09.313288+0000","last_became_peered":"2024-03-30T12:58:09.313288+0000","last_unstale":"2024-03-30T12:58:19.433008+0000","last_undegraded":"2024-03-30T12:58:19.433008+0000","last_fullsized":"2024-03-30T12:58:19.433008+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T12:57:51.514144+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7008420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[1,0,6],"avail_no_missing":["1(0)","0(1)","6(2)","7(0)"],"object_location_counts":[{"shards":"0(1),1(0),6(2)","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":137,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-30T12:58:23.491003+0000","last_change":"2024-03-30T12:57:04.946767+0000","last_active":"2024-03-30T12:58:23.491003+0000","last_peered":"2024-03-30T12:58:23.491003+0000","last_clean":"2024-03-30T12:58:23.491003+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T12:58:23.491003+0000","last_undegraded":"2024-03-30T12:58:23.491003+0000","last_fullsized":"2024-03-30T12:58:23.491003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:57:04.946489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:04.946489+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:26.951070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":155,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-30T12:58:21.880833+0000","last_change":"2024-03-30T12:58:01.131164+0000","last_active":"2024-03-30T12:58:21.880833+0000","last_peered":"2024-03-30T12:58:21.880833+0000","last_clean":"2024-03-30T12:58:21.880833+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T12:58:21.880833+0000","last_undegraded":"2024-03-30T12:58:21.880833+0000","last_fullsized":"2024-03-30T12:58:21.880833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:58:01.130774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:01.130774+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:29.177753+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":114,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-03-30T12:58:20.911518+0000","last_change":"2024-03-30T12:58:13.024975+0000","last_active":"2024-03-30T12:58:20.911518+0000","last_peered":"2024-03-30T12:58:20.911518+0000","last_clean":"2024-03-30T12:58:20.911518+0000","last_became_active":"2024-03-30T12:58:12.369218+0000","last_became_peered":"2024-03-30T12:58:12.369218+0000","last_unstale":"2024-03-30T12:58:20.911518+0000","last_undegraded":"2024-03-30T12:58:20.911518+0000","last_fullsized":"2024-03-30T12:58:20.911518+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T12:58:13.024706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:13.024706+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:36.211866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":138,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T12:58:19.433191+0000","last_change":"2024-03-30T12:56:58.446137+0000","last_active":"2024-03-30T12:58:19.433191+0000","last_peered":"2024-03-30T12:58:19.433191+0000","last_clean":"2024-03-30T12:58:19.433191+0000","last_became_active":"2024-03-30T12:55:43.930722+0000","last_became_peered":"2024-03-30T12:55:43.930722+0000","last_unstale":"2024-03-30T12:58:19.433191+0000","last_undegraded":"2024-03-30T12:58:19.433191+0000","last_fullsized":"2024-03-30T12:58:19.433191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:56:58.445737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:58.445737+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:21.311535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":173,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-30T12:58:21.881028+0000","last_change":"2024-03-30T12:58:11.358195+0000","last_active":"2024-03-30T12:58:21.881028+0000","last_peered":"2024-03-30T12:58:21.881028+0000","last_clean":"2024-03-30T12:58:21.881028+0000","last_became_active":"2024-03-30T12:58:11.357621+0000","last_became_peered":"2024-03-30T12:58:11.357621+0000","last_unstale":"2024-03-30T12:58:21.881028+0000","last_undegraded":"2024-03-30T12:58:21.881028+0000","last_fullsized":"2024-03-30T12:58:21.881028+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:58:04.171994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:04.171994+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:23.627732+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":102,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T12:58:19.434961+0000","last_change":"2024-03-30T12:57:16.932042+0000","last_active":"2024-03-30T12:58:19.434961+0000","last_peered":"2024-03-30T12:58:19.434961+0000","last_clean":"2024-03-30T12:58:19.434961+0000","last_became_active":"2024-03-30T12:57:16.930714+0000","last_became_peered":"2024-03-30T12:57:16.930714+0000","last_unstale":"2024-03-30T12:58:19.434961+0000","last_undegraded":"2024-03-30T12:58:19.434961+0000","last_fullsized":"2024-03-30T12:58:19.434961+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:57:11.300486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:11.300486+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:29.376314+0000","scrub_duration":503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":177,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-30T12:58:19.389618+0000","last_change":"2024-03-30T12:58:19.389708+0000","last_active":"2024-03-30T12:58:19.389618+0000","last_peered":"2024-03-30T12:58:19.389618+0000","last_clean":"2024-03-30T12:58:19.389618+0000","last_became_active":"2024-03-30T12:57:16.930513+0000","last_became_peered":"2024-03-30T12:57:16.930513+0000","last_unstale":"2024-03-30T12:58:19.389618+0000","last_undegraded":"2024-03-30T12:58:19.389618+0000","last_fullsized":"2024-03-30T12:58:19.389618+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:58:19.389431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:19.389431+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:31.361294+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":154,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-30T12:58:18.901237+0000","last_change":"2024-03-30T12:57:11.126991+0000","last_active":"2024-03-30T12:58:18.901237+0000","last_peered":"2024-03-30T12:58:18.901237+0000","last_clean":"2024-03-30T12:58:18.901237+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T12:58:18.901237+0000","last_undegraded":"2024-03-30T12:58:18.901237+0000","last_fullsized":"2024-03-30T12:58:18.901237+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T12:57:11.126678+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:20.211259+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":157,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-30T12:58:23.490551+0000","last_change":"2024-03-30T12:58:14.711329+0000","last_active":"2024-03-30T12:58:23.490551+0000","last_peered":"2024-03-30T12:58:23.490551+0000","last_clean":"2024-03-30T12:58:23.490551+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T12:58:23.490551+0000","last_undegraded":"2024-03-30T12:58:23.490551+0000","last_fullsized":"2024-03-30T12:58:23.490551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:58:14.710869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:14.710869+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:25.694672+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":62,"reported_epoch":76,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-30T12:58:23.490864+0000","last_change":"2024-03-30T12:58:09.319904+0000","last_active":"2024-03-30T12:58:23.490864+0000","last_peered":"2024-03-30T12:58:23.490864+0000","last_clean":"2024-03-30T12:58:06.285343+0000","last_became_active":"2024-03-30T12:58:09.316916+0000","last_became_peered":"2024-03-30T12:58:09.316916+0000","last_unstale":"2024-03-30T12:58:23.490864+0000","last_undegraded":"2024-03-30T12:58:23.490864+0000","last_fullsized":"2024-03-30T12:58:23.490864+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:56:56.870694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:56.870694+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[5,3,2],"avail_no_missing":["5(0)","2(2)","3(1)","7(0)"],"object_location_counts":[{"shards":"2(2),3(1),5(0)","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":157,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-30T12:58:23.490543+0000","last_change":"2024-03-30T12:58:07.734464+0000","last_active":"2024-03-30T12:58:23.490543+0000","last_peered":"2024-03-30T12:58:23.490543+0000","last_clean":"2024-03-30T12:58:23.490543+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T12:58:23.490543+0000","last_undegraded":"2024-03-30T12:58:23.490543+0000","last_fullsized":"2024-03-30T12:58:23.490543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:58:07.734198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:07.734198+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:13.487012+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":134,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-03-30T12:58:19.433006+0000","last_change":"2024-03-30T12:57:09.558059+0000","last_active":"2024-03-30T12:58:19.433006+0000","last_peered":"2024-03-30T12:58:19.433006+0000","last_clean":"2024-03-30T12:58:19.433006+0000","last_became_active":"2024-03-30T12:55:43.925945+0000","last_became_peered":"2024-03-30T12:55:43.925945+0000","last_unstale":"2024-03-30T12:58:19.433006+0000","last_undegraded":"2024-03-30T12:58:19.433006+0000","last_fullsized":"2024-03-30T12:58:19.433006+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:09.557910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:09.557910+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:37.248333+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":97,"num_read_kb":4618,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44621991,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":95780864,"data_stored":95780864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509476,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42936,"kb_used_data":15912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94328904,"statfs":{"total":96636764160,"available":96592797696,"internally_reserved":0,"allocated":16293888,"data_stored":15820338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542181,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47160,"kb_used_data":20136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94324680,"statfs":{"total":96636764160,"available":96588472320,"internally_reserved":0,"allocated":20619264,"data_stored":20112204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822670,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39912,"kb_used_data":12372,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94331928,"statfs":{"total":96636764160,"available":96595894272,"internally_reserved":0,"allocated":12668928,"data_stored":12188676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4399,"internal_metadata":28176081},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509477,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34148,"kb_used_data":7128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337692,"statfs":{"total":96636764160,"available":96601796608,"internally_reserved":0,"allocated":7299072,"data_stored":6825522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509477,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34976,"kb_used_data":7948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94336864,"statfs":{"total":96636764160,"available":96600948736,"internally_reserved":0,"allocated":8138752,"data_stored":7665202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509477,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41320,"kb_used_data":14292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330520,"statfs":{"total":96636764160,"available":96594452480,"internally_reserved":0,"allocated":14635008,"data_stored":14168160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":59,"seq":253403070469,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36492,"kb_used_data":8960,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94335348,"statfs":{"total":96636764160,"available":96599396352,"internally_reserved":0,"allocated":9175040,"data_stored":8688086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4391,"internal_metadata":28176089},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574886,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40360,"kb_used_data":13340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331480,"statfs":{"total":96636764160,"available":96595435520,"internally_reserved":0,"allocated":13660160,"data_stored":13166504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7315456,"data_stored":7315456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19734528,"data_stored":19734528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8327168,"data_stored":8327168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:58:25.433 INFO:tasks.thrashosds.thrasher:Choosing PG 1.3 with acting set [1, 6, 5] 2024-03-30T12:58:25.433 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 1, 2, 3, 5, 6, 4, 7] 2024-03-30T12:58:25.433 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-30T12:58:25.433 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T12:58:25.515 INFO:tasks.ceph.osd.1:Stopped 2024-03-30T12:58:25.516 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 3, 5, 6, 4, 7] 2024-03-30T12:58:25.516 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 1 2024-03-30T12:58:25.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:25.628+0000 7f05cc7af640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:25.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:25.644+0000 7f05cc7af640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:25.832 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T12:58:25.828+0000 7fbb38f08640 -1 mon.a@0(leader).osd e77 definitely_dead 0 2024-03-30T12:58:26.508 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.1. 2024-03-30T12:58:26.535 INFO:tasks.thrashosds.thrasher:dead_osds=[1], live_osds=[0, 2, 3, 5, 6, 4, 7] 2024-03-30T12:58:26.535 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T12:58:26.536 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:58:26.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:26.652+0000 7f5ba0fdb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:26.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:26.664+0000 7f5ba0fdb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:26.885 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:58:26.886 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T12:58:26.906 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-03-30T12:58:26.509802+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":97,"num_read_kb":4618,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44621991,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":46,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":282584,"kb_used_data":92404,"kb_used_omap":16,"kb_used_meta":190063,"kb_avail":660320296,"statfs":{"total":676457349120,"available":676167983104,"internally_reserved":0,"allocated":94621696,"data_stored":91092424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16766,"internal_metadata":194625154},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.681314"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":159,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-30T12:58:24.497637+0000","last_change":"2024-03-30T12:57:16.932276+0000","last_active":"2024-03-30T12:58:24.497637+0000","last_peered":"2024-03-30T12:58:24.497637+0000","last_clean":"2024-03-30T12:58:24.497637+0000","last_became_active":"2024-03-30T12:57:16.931067+0000","last_became_peered":"2024-03-30T12:57:16.931067+0000","last_unstale":"2024-03-30T12:58:24.497637+0000","last_undegraded":"2024-03-30T12:58:24.497637+0000","last_fullsized":"2024-03-30T12:58:24.497637+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:12.143245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:12.143245+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:38.357453+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":128,"reported_epoch":77,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-30T12:58:25.901345+0000","last_change":"2024-03-30T12:58:09.318656+0000","last_active":"2024-03-30T12:58:25.901345+0000","last_peered":"2024-03-30T12:58:25.901345+0000","last_clean":"2024-03-30T12:58:06.286271+0000","last_became_active":"2024-03-30T12:58:09.314481+0000","last_became_peered":"2024-03-30T12:58:09.314481+0000","last_unstale":"2024-03-30T12:58:25.901345+0000","last_undegraded":"2024-03-30T12:58:25.901345+0000","last_fullsized":"2024-03-30T12:58:25.901345+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:56:54.949876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:54.949876+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,1,3],"avail_no_missing":["4(0)","1(1)","3(2)","7(1)"],"object_location_counts":[{"shards":"1(1),3(2),4(0)","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":88,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-30T12:58:25.010910+0000","last_change":"2024-03-30T12:58:23.157256+0000","last_active":"2024-03-30T12:58:25.010910+0000","last_peered":"2024-03-30T12:58:25.010910+0000","last_clean":"2024-03-30T12:58:25.010910+0000","last_became_active":"2024-03-30T12:58:11.361337+0000","last_became_peered":"2024-03-30T12:58:11.361337+0000","last_unstale":"2024-03-30T12:58:25.010910+0000","last_undegraded":"2024-03-30T12:58:25.010910+0000","last_fullsized":"2024-03-30T12:58:25.010910+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:58:23.156870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:23.156870+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:33.109719+0000","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":80,"reported_epoch":77,"state":"stale+active+remapped+backfill_toofull","last_fresh":"2024-03-30T12:58:24.494863+0000","last_change":"2024-03-30T12:58:09.324631+0000","last_active":"2024-03-30T12:58:24.494863+0000","last_peered":"2024-03-30T12:58:24.494863+0000","last_clean":"2024-03-30T12:58:06.282146+0000","last_became_active":"2024-03-30T12:58:09.313288+0000","last_became_peered":"2024-03-30T12:58:09.313288+0000","last_unstale":"2024-03-30T12:58:26.509837+0000","last_undegraded":"2024-03-30T12:58:24.494863+0000","last_fullsized":"2024-03-30T12:58:24.494863+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T12:57:51.514144+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7008420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[1,0,6],"avail_no_missing":["1(0)","0(1)","6(2)","7(0)"],"object_location_counts":[{"shards":"0(1),1(0),6(2)","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":137,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-30T12:58:23.491003+0000","last_change":"2024-03-30T12:57:04.946767+0000","last_active":"2024-03-30T12:58:23.491003+0000","last_peered":"2024-03-30T12:58:23.491003+0000","last_clean":"2024-03-30T12:58:23.491003+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T12:58:23.491003+0000","last_undegraded":"2024-03-30T12:58:23.491003+0000","last_fullsized":"2024-03-30T12:58:23.491003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:57:04.946489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:04.946489+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:26.951070+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":155,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-30T12:58:21.880833+0000","last_change":"2024-03-30T12:58:01.131164+0000","last_active":"2024-03-30T12:58:21.880833+0000","last_peered":"2024-03-30T12:58:21.880833+0000","last_clean":"2024-03-30T12:58:21.880833+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T12:58:21.880833+0000","last_undegraded":"2024-03-30T12:58:21.880833+0000","last_fullsized":"2024-03-30T12:58:21.880833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:58:01.130774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:01.130774+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:29.177753+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":122,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-30T12:58:25.010840+0000","last_change":"2024-03-30T12:58:13.024975+0000","last_active":"2024-03-30T12:58:25.010840+0000","last_peered":"2024-03-30T12:58:25.010840+0000","last_clean":"2024-03-30T12:58:25.010840+0000","last_became_active":"2024-03-30T12:58:12.369218+0000","last_became_peered":"2024-03-30T12:58:12.369218+0000","last_unstale":"2024-03-30T12:58:25.010840+0000","last_undegraded":"2024-03-30T12:58:25.010840+0000","last_fullsized":"2024-03-30T12:58:25.010840+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T12:58:13.024706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:13.024706+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:36.211866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":156,"reported_epoch":77,"state":"stale+active+clean","last_fresh":"2024-03-30T12:58:24.494907+0000","last_change":"2024-03-30T12:58:23.191266+0000","last_active":"2024-03-30T12:58:24.494907+0000","last_peered":"2024-03-30T12:58:24.494907+0000","last_clean":"2024-03-30T12:58:24.494907+0000","last_became_active":"2024-03-30T12:55:43.930722+0000","last_became_peered":"2024-03-30T12:55:43.930722+0000","last_unstale":"2024-03-30T12:58:26.509843+0000","last_undegraded":"2024-03-30T12:58:24.494907+0000","last_fullsized":"2024-03-30T12:58:24.494907+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:58:23.190883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:23.190883+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:27.459512+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":173,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-03-30T12:58:21.881028+0000","last_change":"2024-03-30T12:58:11.358195+0000","last_active":"2024-03-30T12:58:21.881028+0000","last_peered":"2024-03-30T12:58:21.881028+0000","last_clean":"2024-03-30T12:58:21.881028+0000","last_became_active":"2024-03-30T12:58:11.357621+0000","last_became_peered":"2024-03-30T12:58:11.357621+0000","last_unstale":"2024-03-30T12:58:21.881028+0000","last_undegraded":"2024-03-30T12:58:21.881028+0000","last_fullsized":"2024-03-30T12:58:21.881028+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:58:04.171994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:04.171994+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:23.627732+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":112,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-30T12:58:25.901424+0000","last_change":"2024-03-30T12:57:16.932042+0000","last_active":"2024-03-30T12:58:25.901424+0000","last_peered":"2024-03-30T12:58:25.901424+0000","last_clean":"2024-03-30T12:58:25.901424+0000","last_became_active":"2024-03-30T12:57:16.930714+0000","last_became_peered":"2024-03-30T12:57:16.930714+0000","last_unstale":"2024-03-30T12:58:25.901424+0000","last_undegraded":"2024-03-30T12:58:25.901424+0000","last_fullsized":"2024-03-30T12:58:25.901424+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:57:11.300486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:11.300486+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:29.376314+0000","scrub_duration":503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":188,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-30T12:58:24.497753+0000","last_change":"2024-03-30T12:58:19.389708+0000","last_active":"2024-03-30T12:58:24.497753+0000","last_peered":"2024-03-30T12:58:24.497753+0000","last_clean":"2024-03-30T12:58:24.497753+0000","last_became_active":"2024-03-30T12:57:16.930513+0000","last_became_peered":"2024-03-30T12:57:16.930513+0000","last_unstale":"2024-03-30T12:58:24.497753+0000","last_undegraded":"2024-03-30T12:58:24.497753+0000","last_fullsized":"2024-03-30T12:58:24.497753+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:58:19.389431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:19.389431+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:31.361294+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":173,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-30T12:58:24.497988+0000","last_change":"2024-03-30T12:58:23.382393+0000","last_active":"2024-03-30T12:58:24.497988+0000","last_peered":"2024-03-30T12:58:24.497988+0000","last_clean":"2024-03-30T12:58:24.497988+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T12:58:24.497988+0000","last_undegraded":"2024-03-30T12:58:24.497988+0000","last_fullsized":"2024-03-30T12:58:24.497988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T12:58:23.382129+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T12:58:23.382129+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:23.404087+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":157,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-30T12:58:23.490551+0000","last_change":"2024-03-30T12:58:14.711329+0000","last_active":"2024-03-30T12:58:23.490551+0000","last_peered":"2024-03-30T12:58:23.490551+0000","last_clean":"2024-03-30T12:58:23.490551+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T12:58:23.490551+0000","last_undegraded":"2024-03-30T12:58:23.490551+0000","last_fullsized":"2024-03-30T12:58:23.490551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:58:14.710869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:14.710869+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:25.694672+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":62,"reported_epoch":76,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-30T12:58:23.490864+0000","last_change":"2024-03-30T12:58:09.319904+0000","last_active":"2024-03-30T12:58:23.490864+0000","last_peered":"2024-03-30T12:58:23.490864+0000","last_clean":"2024-03-30T12:58:06.285343+0000","last_became_active":"2024-03-30T12:58:09.316916+0000","last_became_peered":"2024-03-30T12:58:09.316916+0000","last_unstale":"2024-03-30T12:58:23.490864+0000","last_undegraded":"2024-03-30T12:58:23.490864+0000","last_fullsized":"2024-03-30T12:58:23.490864+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:56:56.870694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:56.870694+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[5,3,2],"avail_no_missing":["5(0)","2(2)","3(1)","7(0)"],"object_location_counts":[{"shards":"2(2),3(1),5(0)","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":157,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-03-30T12:58:23.490543+0000","last_change":"2024-03-30T12:58:07.734464+0000","last_active":"2024-03-30T12:58:23.490543+0000","last_peered":"2024-03-30T12:58:23.490543+0000","last_clean":"2024-03-30T12:58:23.490543+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T12:58:23.490543+0000","last_undegraded":"2024-03-30T12:58:23.490543+0000","last_fullsized":"2024-03-30T12:58:23.490543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:58:07.734198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:07.734198+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:13.487012+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":144,"reported_epoch":77,"state":"stale+active+clean","last_fresh":"2024-03-30T12:58:24.494883+0000","last_change":"2024-03-30T12:57:09.558059+0000","last_active":"2024-03-30T12:58:24.494883+0000","last_peered":"2024-03-30T12:58:24.494883+0000","last_clean":"2024-03-30T12:58:24.494883+0000","last_became_active":"2024-03-30T12:55:43.925945+0000","last_became_peered":"2024-03-30T12:55:43.925945+0000","last_unstale":"2024-03-30T12:58:26.509847+0000","last_undegraded":"2024-03-30T12:58:24.494883+0000","last_fullsized":"2024-03-30T12:58:24.494883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:09.557910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:57:09.557910+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:58:37.248333+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":97,"num_read_kb":4618,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44621991,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":95780864,"data_stored":95780864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509477,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42992,"kb_used_data":15972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94328848,"statfs":{"total":96636764160,"available":96592740352,"internally_reserved":0,"allocated":16355328,"data_stored":15848268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542181,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47160,"kb_used_data":20136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94324680,"statfs":{"total":96636764160,"available":96588472320,"internally_reserved":0,"allocated":20619264,"data_stored":20112204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822671,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39952,"kb_used_data":12420,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94331888,"statfs":{"total":96636764160,"available":96595853312,"internally_reserved":0,"allocated":12718080,"data_stored":12211020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4399,"internal_metadata":28176081},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509478,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34200,"kb_used_data":7176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337640,"statfs":{"total":96636764160,"available":96601743360,"internally_reserved":0,"allocated":7348224,"data_stored":6847866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509478,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41376,"kb_used_data":14352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330464,"statfs":{"total":96636764160,"available":96594395136,"internally_reserved":0,"allocated":14696448,"data_stored":14196090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":59,"seq":253403070470,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36544,"kb_used_data":9008,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94335296,"statfs":{"total":96636764160,"available":96599343104,"internally_reserved":0,"allocated":9224192,"data_stored":8710472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4391,"internal_metadata":28176089},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574886,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40360,"kb_used_data":13340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331480,"statfs":{"total":96636764160,"available":96595435520,"internally_reserved":0,"allocated":13660160,"data_stored":13166504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7315456,"data_stored":7315456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19734528,"data_stored":19734528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8327168,"data_stored":8327168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:58:26.907 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T12:58:51.907 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T12:58:51.908 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:58:52.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:52.031+0000 7f5c0dd0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:52.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:52.043+0000 7f5c0dd0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:52.221 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:58:52.221 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T12:58:52.238 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":174,"stamp":"2024-03-30T12:58:51.837721+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60655671,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":49,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":289276,"kb_used_data":99120,"kb_used_omap":16,"kb_used_meta":190063,"kb_avail":660313604,"statfs":{"total":676457349120,"available":676161130496,"internally_reserved":0,"allocated":101498880,"data_stored":97716346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16766,"internal_metadata":194625154},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003578"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":175,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T12:58:40.339319+0000","last_change":"2024-03-30T12:58:40.339390+0000","last_active":"2024-03-30T12:58:40.339319+0000","last_peered":"2024-03-30T12:58:40.339319+0000","last_clean":"2024-03-30T12:58:40.339319+0000","last_became_active":"2024-03-30T12:57:16.931067+0000","last_became_peered":"2024-03-30T12:57:16.931067+0000","last_unstale":"2024-03-30T12:58:40.339319+0000","last_undegraded":"2024-03-30T12:58:40.339319+0000","last_fullsized":"2024-03-30T12:58:40.339319+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:58:40.339096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:40.339096+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:53.541716+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":160,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T12:58:30.264842+0000","last_change":"2024-03-30T12:58:30.264910+0000","last_active":"2024-03-30T12:58:30.264842+0000","last_peered":"2024-03-30T12:58:30.264842+0000","last_clean":"2024-03-30T12:58:30.264842+0000","last_became_active":"2024-03-30T12:58:29.543016+0000","last_became_peered":"2024-03-30T12:58:29.543016+0000","last_unstale":"2024-03-30T12:58:30.264842+0000","last_undegraded":"2024-03-30T12:58:30.264842+0000","last_fullsized":"2024-03-30T12:58:30.264842+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T12:58:30.264658+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:50.191800+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":110,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T12:58:29.540672+0000","last_change":"2024-03-30T12:58:28.451419+0000","last_active":"2024-03-30T12:58:29.540672+0000","last_peered":"2024-03-30T12:58:29.540672+0000","last_clean":"2024-03-30T12:58:29.540672+0000","last_became_active":"2024-03-30T12:58:27.533702+0000","last_became_peered":"2024-03-30T12:58:27.533702+0000","last_unstale":"2024-03-30T12:58:29.540672+0000","last_undegraded":"2024-03-30T12:58:29.540672+0000","last_fullsized":"2024-03-30T12:58:29.540672+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:58:23.156870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:23.156870+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:47.705910+0000","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":40,"reported_epoch":81,"state":"active+undersized+remapped+backfill_toofull","last_fresh":"2024-03-30T12:58:29.801905+0000","last_change":"2024-03-30T12:58:27.537810+0000","last_active":"2024-03-30T12:58:29.801905+0000","last_peered":"2024-03-30T12:58:29.801905+0000","last_clean":"2024-03-30T12:55:47.313349+0000","last_became_active":"2024-03-30T12:58:27.535361+0000","last_became_peered":"2024-03-30T12:58:27.535361+0000","last_unstale":"2024-03-30T12:58:29.801905+0000","last_undegraded":"2024-03-30T12:58:29.801905+0000","last_fullsized":"2024-03-30T12:58:27.529062+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T12:57:51.514144+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[2147483647,0,6],"avail_no_missing":["0(1)","6(2)","7(0)"],"object_location_counts":[{"shards":"0(1),6(2)","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":155,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T12:58:33.818811+0000","last_change":"2024-03-30T12:58:33.818975+0000","last_active":"2024-03-30T12:58:33.818811+0000","last_peered":"2024-03-30T12:58:33.818811+0000","last_clean":"2024-03-30T12:58:33.818811+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T12:58:33.818811+0000","last_undegraded":"2024-03-30T12:58:33.818811+0000","last_fullsized":"2024-03-30T12:58:33.818811+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:58:33.818576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:33.818576+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:00.736977+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":169,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T12:58:29.540560+0000","last_change":"2024-03-30T12:58:01.131164+0000","last_active":"2024-03-30T12:58:29.540560+0000","last_peered":"2024-03-30T12:58:29.540560+0000","last_clean":"2024-03-30T12:58:29.540560+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T12:58:29.540560+0000","last_undegraded":"2024-03-30T12:58:29.540560+0000","last_fullsized":"2024-03-30T12:58:29.540560+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:58:01.130774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:01.130774+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:29.177753+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":130,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T12:58:29.540655+0000","last_change":"2024-03-30T12:58:13.024975+0000","last_active":"2024-03-30T12:58:29.540655+0000","last_peered":"2024-03-30T12:58:29.540655+0000","last_clean":"2024-03-30T12:58:29.540655+0000","last_became_active":"2024-03-30T12:58:12.369218+0000","last_became_peered":"2024-03-30T12:58:12.369218+0000","last_unstale":"2024-03-30T12:58:29.540655+0000","last_undegraded":"2024-03-30T12:58:29.540655+0000","last_fullsized":"2024-03-30T12:58:29.540655+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T12:58:13.024706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:13.024706+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:36.211866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T12:58:29.536740+0000","last_change":"2024-03-30T12:58:27.599029+0000","last_active":"2024-03-30T12:58:29.536740+0000","last_peered":"2024-03-30T12:58:29.536740+0000","last_clean":"2024-03-30T12:58:29.536740+0000","last_became_active":"2024-03-30T12:58:27.536475+0000","last_became_peered":"2024-03-30T12:58:27.536475+0000","last_unstale":"2024-03-30T12:58:29.536740+0000","last_undegraded":"2024-03-30T12:58:29.536740+0000","last_fullsized":"2024-03-30T12:58:29.536740+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:58:23.190883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:23.190883+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:50.816015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7344560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":187,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T12:58:29.540879+0000","last_change":"2024-03-30T12:58:11.358195+0000","last_active":"2024-03-30T12:58:29.540879+0000","last_peered":"2024-03-30T12:58:29.540879+0000","last_clean":"2024-03-30T12:58:29.540879+0000","last_became_active":"2024-03-30T12:58:11.357621+0000","last_became_peered":"2024-03-30T12:58:11.357621+0000","last_unstale":"2024-03-30T12:58:29.540879+0000","last_undegraded":"2024-03-30T12:58:29.540879+0000","last_fullsized":"2024-03-30T12:58:29.540879+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:58:04.171994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:04.171994+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:23.627732+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":128,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T12:58:31.220457+0000","last_change":"2024-03-30T12:58:31.220524+0000","last_active":"2024-03-30T12:58:31.220457+0000","last_peered":"2024-03-30T12:58:31.220457+0000","last_clean":"2024-03-30T12:58:31.220457+0000","last_became_active":"2024-03-30T12:57:16.930714+0000","last_became_peered":"2024-03-30T12:57:16.930714+0000","last_unstale":"2024-03-30T12:58:31.220457+0000","last_undegraded":"2024-03-30T12:58:31.220457+0000","last_fullsized":"2024-03-30T12:58:31.220457+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:58:31.220288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:31.220288+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:41.159203+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":196,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T12:58:29.802529+0000","last_change":"2024-03-30T12:58:19.389708+0000","last_active":"2024-03-30T12:58:29.802529+0000","last_peered":"2024-03-30T12:58:29.802529+0000","last_clean":"2024-03-30T12:58:29.802529+0000","last_became_active":"2024-03-30T12:57:16.930513+0000","last_became_peered":"2024-03-30T12:57:16.930513+0000","last_unstale":"2024-03-30T12:58:29.802529+0000","last_undegraded":"2024-03-30T12:58:29.802529+0000","last_fullsized":"2024-03-30T12:58:29.802529+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:58:19.389431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:19.389431+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:31.361294+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":181,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T12:58:29.802626+0000","last_change":"2024-03-30T12:58:23.382393+0000","last_active":"2024-03-30T12:58:29.802626+0000","last_peered":"2024-03-30T12:58:29.802626+0000","last_clean":"2024-03-30T12:58:29.802626+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T12:58:29.802626+0000","last_undegraded":"2024-03-30T12:58:29.802626+0000","last_fullsized":"2024-03-30T12:58:29.802626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T12:58:23.382129+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T12:58:23.382129+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:23.404087+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":167,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T12:58:29.803604+0000","last_change":"2024-03-30T12:58:14.711329+0000","last_active":"2024-03-30T12:58:29.803604+0000","last_peered":"2024-03-30T12:58:29.803604+0000","last_clean":"2024-03-30T12:58:29.803604+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T12:58:29.803604+0000","last_undegraded":"2024-03-30T12:58:29.803604+0000","last_fullsized":"2024-03-30T12:58:29.803604+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:58:14.710869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:14.710869+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:25.694672+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":76,"reported_epoch":81,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-30T12:58:39.321844+0000","last_change":"2024-03-30T12:58:39.321844+0000","last_active":"2024-03-30T12:58:39.321844+0000","last_peered":"2024-03-30T12:58:39.321844+0000","last_clean":"2024-03-30T12:58:06.285343+0000","last_became_active":"2024-03-30T12:58:09.316916+0000","last_became_peered":"2024-03-30T12:58:09.316916+0000","last_unstale":"2024-03-30T12:58:39.321844+0000","last_undegraded":"2024-03-30T12:58:39.321844+0000","last_fullsized":"2024-03-30T12:58:39.321844+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:56:56.870694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:56.870694+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[5,3,2],"avail_no_missing":["5(0)","2(2)","3(1)","7(0)"],"object_location_counts":[{"shards":"2(2),3(1),5(0)","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":167,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T12:58:29.803709+0000","last_change":"2024-03-30T12:58:07.734464+0000","last_active":"2024-03-30T12:58:29.803709+0000","last_peered":"2024-03-30T12:58:29.803709+0000","last_clean":"2024-03-30T12:58:29.803709+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T12:58:29.803709+0000","last_undegraded":"2024-03-30T12:58:29.803709+0000","last_fullsized":"2024-03-30T12:58:29.803709+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:58:07.734198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:07.734198+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:13.487012+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-30T12:58:34.876179+0000","last_change":"2024-03-30T12:58:34.876355+0000","last_active":"2024-03-30T12:58:34.876179+0000","last_peered":"2024-03-30T12:58:34.876179+0000","last_clean":"2024-03-30T12:58:34.876179+0000","last_became_active":"2024-03-30T12:58:27.535921+0000","last_became_peered":"2024-03-30T12:58:27.535921+0000","last_unstale":"2024-03-30T12:58:34.876179+0000","last_undegraded":"2024-03-30T12:58:34.876179+0000","last_fullsized":"2024-03-30T12:58:34.876179+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:58:34.875943+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:58:34.875943+0000","last_clean_scrub_stamp":"2024-03-30T12:58:34.875943+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:47.189950+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5226911,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60655671,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102191104,"data_stored":102191104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509482,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43048,"kb_used_data":16032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94328792,"statfs":{"total":96636764160,"available":96592683008,"internally_reserved":0,"allocated":16416768,"data_stored":15876406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542186,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47212,"kb_used_data":20196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94324628,"statfs":{"total":96636764160,"available":96588419072,"internally_reserved":0,"allocated":20680704,"data_stored":20140342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822676,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41964,"kb_used_data":14432,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94329876,"statfs":{"total":96636764160,"available":96593793024,"internally_reserved":0,"allocated":14778368,"data_stored":14238006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4399,"internal_metadata":28176081},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509483,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37912,"kb_used_data":10896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333928,"statfs":{"total":96636764160,"available":96597942272,"internally_reserved":0,"allocated":11157504,"data_stored":10617142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509483,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41440,"kb_used_data":14424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330400,"statfs":{"total":96636764160,"available":96594329600,"internally_reserved":0,"allocated":14770176,"data_stored":14229814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":59,"seq":253403070475,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37248,"kb_used_data":9716,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94334592,"statfs":{"total":96636764160,"available":96598622208,"internally_reserved":0,"allocated":9949184,"data_stored":9408822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4391,"internal_metadata":28176089},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574891,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40452,"kb_used_data":13424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331388,"statfs":{"total":96636764160,"available":96595341312,"internally_reserved":0,"allocated":13746176,"data_stored":13205814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7315456,"data_stored":7315456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10211328,"data_stored":10211328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":13832192,"data_stored":13832192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19734528,"data_stored":19734528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9003008,"data_stored":9003008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:58:52.239 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T12:58:52.239 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T12:58:52.239 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-03-30T12:58:52.239 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-30T12:58:52.239 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-30T12:58:52.242 INFO:tasks.ceph.osd.1:Started 2024-03-30T12:58:52.242 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-30T12:58:52.299 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T12:58:52.295+0000 7f435fbe97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:52.307 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T12:58:52.303+0000 7f435fbe97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:52.307 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T12:58:52.303+0000 7f435fbe97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:52.357 INFO:teuthology.orchestra.run.smithi067.stderr:no valid command found; 10 closest matches: 2024-03-30T12:58:52.357 INFO:teuthology.orchestra.run.smithi067.stderr:0 2024-03-30T12:58:52.357 INFO:teuthology.orchestra.run.smithi067.stderr:1 2024-03-30T12:58:52.357 INFO:teuthology.orchestra.run.smithi067.stderr:2 2024-03-30T12:58:52.357 INFO:teuthology.orchestra.run.smithi067.stderr:abort 2024-03-30T12:58:52.357 INFO:teuthology.orchestra.run.smithi067.stderr:assert 2024-03-30T12:58:52.358 INFO:teuthology.orchestra.run.smithi067.stderr:config diff 2024-03-30T12:58:52.358 INFO:teuthology.orchestra.run.smithi067.stderr:config diff get 2024-03-30T12:58:52.358 INFO:teuthology.orchestra.run.smithi067.stderr:config get 2024-03-30T12:58:52.358 INFO:teuthology.orchestra.run.smithi067.stderr:config help [] 2024-03-30T12:58:52.358 INFO:teuthology.orchestra.run.smithi067.stderr:config set ... 2024-03-30T12:58:52.358 INFO:teuthology.orchestra.run.smithi067.stderr:admin_socket: invalid command 2024-03-30T12:58:52.359 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T12:58:52.360 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-30T12:58:52.848 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T12:58:52.843+0000 7f435fbe97c0 -1 Falling back to public interface 2024-03-30T12:58:56.618 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T12:58:56.615+0000 7f435fbe97c0 -1 osd.1 77 log_to_monitors true 2024-03-30T12:58:56.971 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T12:58:56.967+0000 7f43579be640 -1 osd.1 77 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T12:58:57.361 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-30T12:58:57.493 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-30T12:58:57.493 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-30T12:58:57.493 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-03-30T12:58:57.493 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-30T12:58:57.507 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T12:58:57.508 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-03-30T12:58:57.508 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 1 2024-03-30T12:58:57.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:57.615+0000 7fc751f5c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:57.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:57.631+0000 7fc751f5c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:57.816 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T12:58:57.811+0000 7fbb38f08640 -1 mon.a@0(leader).osd e82 definitely_dead 0 2024-03-30T12:58:57.953 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.1. 2024-03-30T12:58:57.979 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-03-30T12:58:57.979 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T12:58:58.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:58.095+0000 7f895c5b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:58.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:58.111+0000 7f895c5b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:58.955 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T12:58:58.984 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T12:58:59.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:59.099+0000 7f0628763640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:59.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:58:59.115+0000 7f0628763640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:58:59.970 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T12:58:59.998 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T12:59:00.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:00.115+0000 7fe2359ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:00.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:00.135+0000 7fe2359ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:00.986 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T12:59:01.010 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T12:59:01.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:01.123+0000 7fd68108f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:01.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:01.143+0000 7fd68108f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:02.013 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T12:59:02.039 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T12:59:02.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:02.155+0000 7f4b44e97640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:02.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:02.171+0000 7f4b44e97640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:03.004 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T12:59:03.028 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T12:59:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:03.143+0000 7f3763aea640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:03.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:03.159+0000 7f3763aea640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:04.012 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T12:59:04.038 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T12:59:04.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:04.155+0000 7f99f8639640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:04.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:04.171+0000 7f99f8639640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:05.017 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T12:59:05.043 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T12:59:05.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:05.159+0000 7f4e4402e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:05.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:05.175+0000 7f4e4402e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:06.032 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T12:59:06.059 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T12:59:06.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:06.175+0000 7f61ad395640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:06.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:06.191+0000 7f61ad395640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T12:59:07.068 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T12:59:07.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:07.183+0000 7f44dbc5a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:07.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:07.199+0000 7f44dbc5a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:08.049 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T12:59:08.074 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T12:59:08.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:08.191+0000 7f2a47ecb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:08.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:08.207+0000 7f2a47ecb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:09.058 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T12:59:09.084 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T12:59:09.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:09.199+0000 7f84c9359640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:09.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:09.215+0000 7f84c9359640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:10.079 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T12:59:10.109 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T12:59:10.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:10.223+0000 7fa9375fb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:10.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:10.239+0000 7fa9375fb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:11.076 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T12:59:11.103 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T12:59:11.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:11.215+0000 7fcb884fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:11.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:11.231+0000 7fcb884fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:12.099 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T12:59:12.125 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T12:59:12.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:12.243+0000 7f591e24e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:12.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:12.259+0000 7f591e24e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T12:59:13.134 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T12:59:13.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:13.251+0000 7fe3ed723640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:13.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:13.267+0000 7fe3ed723640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:14.114 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T12:59:14.140 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T12:59:14.141 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:59:14.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:14.259+0000 7f3b3c8e8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:14.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:14.271+0000 7f3b3c8e8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:14.449 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:59:14.450 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T12:59:14.467 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2024-03-30T12:59:14.116554+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":55109427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":49,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":319304,"kb_used_data":101568,"kb_used_omap":20,"kb_used_meta":217579,"kb_avail":754655416,"statfs":{"total":773094113280,"available":772767145984,"internally_reserved":0,"allocated":104005632,"data_stored":98937872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21164,"internal_metadata":222801236},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.038170"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":198,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T12:59:09.062737+0000","last_change":"2024-03-30T12:58:40.339390+0000","last_active":"2024-03-30T12:59:09.062737+0000","last_peered":"2024-03-30T12:59:09.062737+0000","last_clean":"2024-03-30T12:59:09.062737+0000","last_became_active":"2024-03-30T12:57:16.931067+0000","last_became_peered":"2024-03-30T12:57:16.931067+0000","last_unstale":"2024-03-30T12:59:09.062737+0000","last_undegraded":"2024-03-30T12:59:09.062737+0000","last_fullsized":"2024-03-30T12:59:09.062737+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:58:40.339096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:40.339096+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:53.541716+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":187,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-30T12:59:10.082184+0000","last_change":"2024-03-30T12:58:30.264910+0000","last_active":"2024-03-30T12:59:10.082184+0000","last_peered":"2024-03-30T12:59:10.082184+0000","last_clean":"2024-03-30T12:59:10.082184+0000","last_became_active":"2024-03-30T12:58:29.543016+0000","last_became_peered":"2024-03-30T12:58:29.543016+0000","last_unstale":"2024-03-30T12:59:10.082184+0000","last_undegraded":"2024-03-30T12:59:10.082184+0000","last_fullsized":"2024-03-30T12:59:10.082184+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T12:58:30.264658+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:50.191800+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":145,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-30T12:59:11.081745+0000","last_change":"2024-03-30T12:58:58.961161+0000","last_active":"2024-03-30T12:59:11.081745+0000","last_peered":"2024-03-30T12:59:11.081745+0000","last_clean":"2024-03-30T12:59:11.081745+0000","last_became_active":"2024-03-30T12:58:58.960224+0000","last_became_peered":"2024-03-30T12:58:58.960224+0000","last_unstale":"2024-03-30T12:59:11.081745+0000","last_undegraded":"2024-03-30T12:59:11.081745+0000","last_fullsized":"2024-03-30T12:59:11.081745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:58:23.156870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:23.156870+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:37.567491+0000","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":80,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-30T12:59:11.081724+0000","last_change":"2024-03-30T12:58:59.978677+0000","last_active":"2024-03-30T12:59:11.081724+0000","last_peered":"2024-03-30T12:59:11.081724+0000","last_clean":"2024-03-30T12:59:11.081724+0000","last_became_active":"2024-03-30T12:58:59.977351+0000","last_became_peered":"2024-03-30T12:58:59.977351+0000","last_unstale":"2024-03-30T12:59:11.081724+0000","last_undegraded":"2024-03-30T12:59:11.081724+0000","last_fullsized":"2024-03-30T12:59:11.081724+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T12:57:51.514144+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:14.677574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":188,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-30T12:59:13.952536+0000","last_change":"2024-03-30T12:58:33.818975+0000","last_active":"2024-03-30T12:59:13.952536+0000","last_peered":"2024-03-30T12:59:13.952536+0000","last_clean":"2024-03-30T12:59:13.952536+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T12:59:13.952536+0000","last_undegraded":"2024-03-30T12:59:13.952536+0000","last_fullsized":"2024-03-30T12:59:13.952536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:58:33.818576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:33.818576+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:00.736977+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":199,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-30T12:59:12.098249+0000","last_change":"2024-03-30T12:58:01.131164+0000","last_active":"2024-03-30T12:59:12.098249+0000","last_peered":"2024-03-30T12:59:12.098249+0000","last_clean":"2024-03-30T12:59:12.098249+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T12:59:12.098249+0000","last_undegraded":"2024-03-30T12:59:12.098249+0000","last_fullsized":"2024-03-30T12:59:12.098249+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:58:01.130774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:01.130774+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:29.177753+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":159,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-30T12:59:11.081702+0000","last_change":"2024-03-30T12:58:13.024975+0000","last_active":"2024-03-30T12:59:11.081702+0000","last_peered":"2024-03-30T12:59:11.081702+0000","last_clean":"2024-03-30T12:59:11.081702+0000","last_became_active":"2024-03-30T12:58:12.369218+0000","last_became_peered":"2024-03-30T12:58:12.369218+0000","last_unstale":"2024-03-30T12:59:11.081702+0000","last_undegraded":"2024-03-30T12:59:11.081702+0000","last_fullsized":"2024-03-30T12:59:11.081702+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T12:58:13.024706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:13.024706+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:36.211866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":190,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-30T12:59:12.097659+0000","last_change":"2024-03-30T12:58:59.114692+0000","last_active":"2024-03-30T12:59:12.097659+0000","last_peered":"2024-03-30T12:59:12.097659+0000","last_clean":"2024-03-30T12:59:12.097659+0000","last_became_active":"2024-03-30T12:58:59.114041+0000","last_became_peered":"2024-03-30T12:58:59.114041+0000","last_unstale":"2024-03-30T12:59:12.097659+0000","last_undegraded":"2024-03-30T12:59:12.097659+0000","last_fullsized":"2024-03-30T12:59:12.097659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:58:23.190883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:23.190883+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:49.009057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7344560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":217,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-30T12:59:12.098261+0000","last_change":"2024-03-30T12:58:11.358195+0000","last_active":"2024-03-30T12:59:12.098261+0000","last_peered":"2024-03-30T12:59:12.098261+0000","last_clean":"2024-03-30T12:59:12.098261+0000","last_became_active":"2024-03-30T12:58:11.357621+0000","last_became_peered":"2024-03-30T12:58:11.357621+0000","last_unstale":"2024-03-30T12:59:12.098261+0000","last_undegraded":"2024-03-30T12:59:12.098261+0000","last_fullsized":"2024-03-30T12:59:12.098261+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:58:04.171994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:04.171994+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:23.627732+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":155,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-30T12:59:10.082192+0000","last_change":"2024-03-30T12:58:31.220524+0000","last_active":"2024-03-30T12:59:10.082192+0000","last_peered":"2024-03-30T12:59:10.082192+0000","last_clean":"2024-03-30T12:59:10.082192+0000","last_became_active":"2024-03-30T12:57:16.930714+0000","last_became_peered":"2024-03-30T12:57:16.930714+0000","last_unstale":"2024-03-30T12:59:10.082192+0000","last_undegraded":"2024-03-30T12:59:10.082192+0000","last_fullsized":"2024-03-30T12:59:10.082192+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:58:31.220288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:31.220288+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:41.159203+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":219,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T12:59:09.062778+0000","last_change":"2024-03-30T12:58:19.389708+0000","last_active":"2024-03-30T12:59:09.062778+0000","last_peered":"2024-03-30T12:59:09.062778+0000","last_clean":"2024-03-30T12:59:09.062778+0000","last_became_active":"2024-03-30T12:57:16.930513+0000","last_became_peered":"2024-03-30T12:57:16.930513+0000","last_unstale":"2024-03-30T12:59:09.062778+0000","last_undegraded":"2024-03-30T12:59:09.062778+0000","last_fullsized":"2024-03-30T12:59:09.062778+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:58:19.389431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:19.389431+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:31.361294+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":204,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T12:59:09.063339+0000","last_change":"2024-03-30T12:58:23.382393+0000","last_active":"2024-03-30T12:59:09.063339+0000","last_peered":"2024-03-30T12:59:09.063339+0000","last_clean":"2024-03-30T12:59:09.063339+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T12:59:09.063339+0000","last_undegraded":"2024-03-30T12:59:09.063339+0000","last_fullsized":"2024-03-30T12:59:09.063339+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T12:58:23.382129+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T12:58:23.382129+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:23.404087+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":200,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-30T12:59:13.952530+0000","last_change":"2024-03-30T12:58:14.711329+0000","last_active":"2024-03-30T12:59:13.952530+0000","last_peered":"2024-03-30T12:59:13.952530+0000","last_clean":"2024-03-30T12:59:13.952530+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T12:59:13.952530+0000","last_undegraded":"2024-03-30T12:59:13.952530+0000","last_fullsized":"2024-03-30T12:59:13.952530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:58:14.710869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:14.710869+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:25.694672+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":117,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-30T12:59:11.088606+0000","last_change":"2024-03-30T12:59:11.088606+0000","last_active":"2024-03-30T12:59:11.088606+0000","last_peered":"2024-03-30T12:59:11.088606+0000","last_clean":"2024-03-30T12:59:11.088606+0000","last_became_active":"2024-03-30T12:59:11.087652+0000","last_became_peered":"2024-03-30T12:59:11.087652+0000","last_unstale":"2024-03-30T12:59:11.088606+0000","last_undegraded":"2024-03-30T12:59:11.088606+0000","last_fullsized":"2024-03-30T12:59:11.088606+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:56:56.870694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:56.870694+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":208,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-30T12:59:13.956389+0000","last_change":"2024-03-30T12:59:13.956465+0000","last_active":"2024-03-30T12:59:13.956389+0000","last_peered":"2024-03-30T12:59:13.956389+0000","last_clean":"2024-03-30T12:59:13.956389+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T12:59:13.956389+0000","last_undegraded":"2024-03-30T12:59:13.956389+0000","last_fullsized":"2024-03-30T12:59:13.956389+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:59:13.956186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:13.956186+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:35.452148+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":180,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-30T12:59:12.097679+0000","last_change":"2024-03-30T12:58:59.113868+0000","last_active":"2024-03-30T12:59:12.097679+0000","last_peered":"2024-03-30T12:59:12.097679+0000","last_clean":"2024-03-30T12:59:12.097679+0000","last_became_active":"2024-03-30T12:58:59.113215+0000","last_became_peered":"2024-03-30T12:58:59.113215+0000","last_unstale":"2024-03-30T12:59:12.097679+0000","last_undegraded":"2024-03-30T12:59:12.097679+0000","last_fullsized":"2024-03-30T12:59:12.097679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:58:34.875943+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:58:34.875943+0000","last_clean_scrub_stamp":"2024-03-30T12:58:34.875943+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:01.009498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":55109427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":95059968,"data_stored":95059968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509486,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43208,"kb_used_data":16188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94328632,"statfs":{"total":96636764160,"available":96592519168,"internally_reserved":0,"allocated":16576512,"data_stored":15949840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542191,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46776,"kb_used_data":19748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94325064,"statfs":{"total":96636764160,"available":96588865536,"internally_reserved":0,"allocated":20221952,"data_stored":19574914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822680,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40168,"kb_used_data":12636,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94331672,"statfs":{"total":96636764160,"available":96595632128,"internally_reserved":0,"allocated":12939264,"data_stored":12312592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4399,"internal_metadata":28176081},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509487,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34404,"kb_used_data":7380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337436,"statfs":{"total":96636764160,"available":96601534464,"internally_reserved":0,"allocated":7557120,"data_stored":6943934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285573,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33816,"kb_used_data":6284,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94338024,"statfs":{"total":96636764160,"available":96602136576,"internally_reserved":0,"allocated":6434816,"data_stored":5787778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":28176082},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509487,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41612,"kb_used_data":14580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330228,"statfs":{"total":96636764160,"available":96594153472,"internally_reserved":0,"allocated":14929920,"data_stored":14303248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":59,"seq":253403070479,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38676,"kb_used_data":11136,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94333164,"statfs":{"total":96636764160,"available":96597159936,"internally_reserved":0,"allocated":11403264,"data_stored":10769820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4391,"internal_metadata":28176089},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574896,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40644,"kb_used_data":13616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331196,"statfs":{"total":96636764160,"available":96595144704,"internally_reserved":0,"allocated":13942784,"data_stored":13295746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19079168,"data_stored":19079168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:59:14.468 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:59:14.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:14.579+0000 7f26841e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:14.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:14.591+0000 7f26841e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:14.771 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:59:14.771 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T12:59:14.788 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2024-03-30T12:59:14.116554+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":55109427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":49,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":319304,"kb_used_data":101568,"kb_used_omap":20,"kb_used_meta":217579,"kb_avail":754655416,"statfs":{"total":773094113280,"available":772767145984,"internally_reserved":0,"allocated":104005632,"data_stored":98937872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21164,"internal_metadata":222801236},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.038170"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":198,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T12:59:09.062737+0000","last_change":"2024-03-30T12:58:40.339390+0000","last_active":"2024-03-30T12:59:09.062737+0000","last_peered":"2024-03-30T12:59:09.062737+0000","last_clean":"2024-03-30T12:59:09.062737+0000","last_became_active":"2024-03-30T12:57:16.931067+0000","last_became_peered":"2024-03-30T12:57:16.931067+0000","last_unstale":"2024-03-30T12:59:09.062737+0000","last_undegraded":"2024-03-30T12:59:09.062737+0000","last_fullsized":"2024-03-30T12:59:09.062737+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:58:40.339096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:40.339096+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:53.541716+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":187,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-30T12:59:10.082184+0000","last_change":"2024-03-30T12:58:30.264910+0000","last_active":"2024-03-30T12:59:10.082184+0000","last_peered":"2024-03-30T12:59:10.082184+0000","last_clean":"2024-03-30T12:59:10.082184+0000","last_became_active":"2024-03-30T12:58:29.543016+0000","last_became_peered":"2024-03-30T12:58:29.543016+0000","last_unstale":"2024-03-30T12:59:10.082184+0000","last_undegraded":"2024-03-30T12:59:10.082184+0000","last_fullsized":"2024-03-30T12:59:10.082184+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T12:58:30.264658+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:50.191800+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":145,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-30T12:59:11.081745+0000","last_change":"2024-03-30T12:58:58.961161+0000","last_active":"2024-03-30T12:59:11.081745+0000","last_peered":"2024-03-30T12:59:11.081745+0000","last_clean":"2024-03-30T12:59:11.081745+0000","last_became_active":"2024-03-30T12:58:58.960224+0000","last_became_peered":"2024-03-30T12:58:58.960224+0000","last_unstale":"2024-03-30T12:59:11.081745+0000","last_undegraded":"2024-03-30T12:59:11.081745+0000","last_fullsized":"2024-03-30T12:59:11.081745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:58:23.156870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:23.156870+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:37.567491+0000","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":80,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-30T12:59:11.081724+0000","last_change":"2024-03-30T12:58:59.978677+0000","last_active":"2024-03-30T12:59:11.081724+0000","last_peered":"2024-03-30T12:59:11.081724+0000","last_clean":"2024-03-30T12:59:11.081724+0000","last_became_active":"2024-03-30T12:58:59.977351+0000","last_became_peered":"2024-03-30T12:58:59.977351+0000","last_unstale":"2024-03-30T12:59:11.081724+0000","last_undegraded":"2024-03-30T12:59:11.081724+0000","last_fullsized":"2024-03-30T12:59:11.081724+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T12:57:51.514144+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:14.677574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":188,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-30T12:59:13.952536+0000","last_change":"2024-03-30T12:58:33.818975+0000","last_active":"2024-03-30T12:59:13.952536+0000","last_peered":"2024-03-30T12:59:13.952536+0000","last_clean":"2024-03-30T12:59:13.952536+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T12:59:13.952536+0000","last_undegraded":"2024-03-30T12:59:13.952536+0000","last_fullsized":"2024-03-30T12:59:13.952536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:58:33.818576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:33.818576+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:00.736977+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":199,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-30T12:59:12.098249+0000","last_change":"2024-03-30T12:58:01.131164+0000","last_active":"2024-03-30T12:59:12.098249+0000","last_peered":"2024-03-30T12:59:12.098249+0000","last_clean":"2024-03-30T12:59:12.098249+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T12:59:12.098249+0000","last_undegraded":"2024-03-30T12:59:12.098249+0000","last_fullsized":"2024-03-30T12:59:12.098249+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:58:01.130774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:01.130774+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:29.177753+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":159,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-30T12:59:11.081702+0000","last_change":"2024-03-30T12:58:13.024975+0000","last_active":"2024-03-30T12:59:11.081702+0000","last_peered":"2024-03-30T12:59:11.081702+0000","last_clean":"2024-03-30T12:59:11.081702+0000","last_became_active":"2024-03-30T12:58:12.369218+0000","last_became_peered":"2024-03-30T12:58:12.369218+0000","last_unstale":"2024-03-30T12:59:11.081702+0000","last_undegraded":"2024-03-30T12:59:11.081702+0000","last_fullsized":"2024-03-30T12:59:11.081702+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T12:58:13.024706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:13.024706+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:36.211866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":190,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-30T12:59:12.097659+0000","last_change":"2024-03-30T12:58:59.114692+0000","last_active":"2024-03-30T12:59:12.097659+0000","last_peered":"2024-03-30T12:59:12.097659+0000","last_clean":"2024-03-30T12:59:12.097659+0000","last_became_active":"2024-03-30T12:58:59.114041+0000","last_became_peered":"2024-03-30T12:58:59.114041+0000","last_unstale":"2024-03-30T12:59:12.097659+0000","last_undegraded":"2024-03-30T12:59:12.097659+0000","last_fullsized":"2024-03-30T12:59:12.097659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:58:23.190883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:23.190883+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:49.009057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7344560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":217,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-30T12:59:12.098261+0000","last_change":"2024-03-30T12:58:11.358195+0000","last_active":"2024-03-30T12:59:12.098261+0000","last_peered":"2024-03-30T12:59:12.098261+0000","last_clean":"2024-03-30T12:59:12.098261+0000","last_became_active":"2024-03-30T12:58:11.357621+0000","last_became_peered":"2024-03-30T12:58:11.357621+0000","last_unstale":"2024-03-30T12:59:12.098261+0000","last_undegraded":"2024-03-30T12:59:12.098261+0000","last_fullsized":"2024-03-30T12:59:12.098261+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:58:04.171994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:04.171994+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:23.627732+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":155,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-30T12:59:10.082192+0000","last_change":"2024-03-30T12:58:31.220524+0000","last_active":"2024-03-30T12:59:10.082192+0000","last_peered":"2024-03-30T12:59:10.082192+0000","last_clean":"2024-03-30T12:59:10.082192+0000","last_became_active":"2024-03-30T12:57:16.930714+0000","last_became_peered":"2024-03-30T12:57:16.930714+0000","last_unstale":"2024-03-30T12:59:10.082192+0000","last_undegraded":"2024-03-30T12:59:10.082192+0000","last_fullsized":"2024-03-30T12:59:10.082192+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:58:31.220288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:31.220288+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:41.159203+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":219,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T12:59:09.062778+0000","last_change":"2024-03-30T12:58:19.389708+0000","last_active":"2024-03-30T12:59:09.062778+0000","last_peered":"2024-03-30T12:59:09.062778+0000","last_clean":"2024-03-30T12:59:09.062778+0000","last_became_active":"2024-03-30T12:57:16.930513+0000","last_became_peered":"2024-03-30T12:57:16.930513+0000","last_unstale":"2024-03-30T12:59:09.062778+0000","last_undegraded":"2024-03-30T12:59:09.062778+0000","last_fullsized":"2024-03-30T12:59:09.062778+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:58:19.389431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:19.389431+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:31.361294+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":204,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-30T12:59:09.063339+0000","last_change":"2024-03-30T12:58:23.382393+0000","last_active":"2024-03-30T12:59:09.063339+0000","last_peered":"2024-03-30T12:59:09.063339+0000","last_clean":"2024-03-30T12:59:09.063339+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T12:59:09.063339+0000","last_undegraded":"2024-03-30T12:59:09.063339+0000","last_fullsized":"2024-03-30T12:59:09.063339+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T12:58:23.382129+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T12:58:23.382129+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:23.404087+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":200,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-30T12:59:13.952530+0000","last_change":"2024-03-30T12:58:14.711329+0000","last_active":"2024-03-30T12:59:13.952530+0000","last_peered":"2024-03-30T12:59:13.952530+0000","last_clean":"2024-03-30T12:59:13.952530+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T12:59:13.952530+0000","last_undegraded":"2024-03-30T12:59:13.952530+0000","last_fullsized":"2024-03-30T12:59:13.952530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:58:14.710869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:14.710869+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:25.694672+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":117,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-30T12:59:11.088606+0000","last_change":"2024-03-30T12:59:11.088606+0000","last_active":"2024-03-30T12:59:11.088606+0000","last_peered":"2024-03-30T12:59:11.088606+0000","last_clean":"2024-03-30T12:59:11.088606+0000","last_became_active":"2024-03-30T12:59:11.087652+0000","last_became_peered":"2024-03-30T12:59:11.087652+0000","last_unstale":"2024-03-30T12:59:11.088606+0000","last_undegraded":"2024-03-30T12:59:11.088606+0000","last_fullsized":"2024-03-30T12:59:11.088606+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:56:56.870694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:56.870694+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":208,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-30T12:59:13.956389+0000","last_change":"2024-03-30T12:59:13.956465+0000","last_active":"2024-03-30T12:59:13.956389+0000","last_peered":"2024-03-30T12:59:13.956389+0000","last_clean":"2024-03-30T12:59:13.956389+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T12:59:13.956389+0000","last_undegraded":"2024-03-30T12:59:13.956389+0000","last_fullsized":"2024-03-30T12:59:13.956389+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:59:13.956186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:13.956186+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:35.452148+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":180,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-30T12:59:12.097679+0000","last_change":"2024-03-30T12:58:59.113868+0000","last_active":"2024-03-30T12:59:12.097679+0000","last_peered":"2024-03-30T12:59:12.097679+0000","last_clean":"2024-03-30T12:59:12.097679+0000","last_became_active":"2024-03-30T12:58:59.113215+0000","last_became_peered":"2024-03-30T12:58:59.113215+0000","last_unstale":"2024-03-30T12:59:12.097679+0000","last_undegraded":"2024-03-30T12:59:12.097679+0000","last_fullsized":"2024-03-30T12:59:12.097679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:58:34.875943+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:58:34.875943+0000","last_clean_scrub_stamp":"2024-03-30T12:58:34.875943+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:01.009498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":55109427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":95059968,"data_stored":95059968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509486,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43208,"kb_used_data":16188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94328632,"statfs":{"total":96636764160,"available":96592519168,"internally_reserved":0,"allocated":16576512,"data_stored":15949840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542191,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46776,"kb_used_data":19748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94325064,"statfs":{"total":96636764160,"available":96588865536,"internally_reserved":0,"allocated":20221952,"data_stored":19574914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822680,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40168,"kb_used_data":12636,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94331672,"statfs":{"total":96636764160,"available":96595632128,"internally_reserved":0,"allocated":12939264,"data_stored":12312592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4399,"internal_metadata":28176081},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509487,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34404,"kb_used_data":7380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337436,"statfs":{"total":96636764160,"available":96601534464,"internally_reserved":0,"allocated":7557120,"data_stored":6943934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285573,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33816,"kb_used_data":6284,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94338024,"statfs":{"total":96636764160,"available":96602136576,"internally_reserved":0,"allocated":6434816,"data_stored":5787778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":28176082},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509487,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41612,"kb_used_data":14580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330228,"statfs":{"total":96636764160,"available":96594153472,"internally_reserved":0,"allocated":14929920,"data_stored":14303248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":59,"seq":253403070479,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38676,"kb_used_data":11136,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94333164,"statfs":{"total":96636764160,"available":96597159936,"internally_reserved":0,"allocated":11403264,"data_stored":10769820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4391,"internal_metadata":28176089},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574896,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40644,"kb_used_data":13616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331196,"statfs":{"total":96636764160,"available":96595144704,"internally_reserved":0,"allocated":13942784,"data_stored":13295746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19079168,"data_stored":19079168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:59:14.789 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T12:59:14.790 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T12:59:14.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:14.891+0000 7fa0e85a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:14.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:14.903+0000 7fa0e85a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:15.080 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:59:15.081 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":99,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T12:59:14.104385+0000","last_up_change":"2024-03-30T12:58:57.941325+0000","last_in_change":"2024-03-30T12:58:57.941325+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":77,"up_from":83,"up_thru":83,"down_at":78,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6818","nonce":3104224893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6821","nonce":3104224893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6828","nonce":3104224893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6825","nonce":3104224893}]},"public_addr":"172.21.15.67:6818/3104224893","cluster_addr":"172.21.15.67:6821/3104224893","heartbeat_back_addr":"172.21.15.67:6828/3104224893","heartbeat_front_addr":"172.21.15.67:6825/3104224893","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":78,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6801","nonce":1504348724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6803","nonce":1504348724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6807","nonce":1504348724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6805","nonce":1504348724}]},"public_addr":"172.21.15.67:6801/1504348724","cluster_addr":"172.21.15.67:6803/1504348724","heartbeat_back_addr":"172.21.15.67:6807/1504348724","heartbeat_front_addr":"172.21.15.67:6805/1504348724","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":25,"up_from":36,"up_thru":80,"down_at":26,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1927174653},{"type":"v1","addr":"172.21.15.82:6803","nonce":1927174653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":1927174653},{"type":"v1","addr":"172.21.15.82:6807","nonce":1927174653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":1927174653},{"type":"v1","addr":"172.21.15.82:6815","nonce":1927174653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":1927174653},{"type":"v1","addr":"172.21.15.82:6811","nonce":1927174653}]},"public_addr":"172.21.15.82:6803/1927174653","cluster_addr":"172.21.15.82:6807/1927174653","heartbeat_back_addr":"172.21.15.82:6815/1927174653","heartbeat_front_addr":"172.21.15.82:6811/1927174653","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6817","nonce":3467856022}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6819","nonce":3467856022}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6823","nonce":3467856022}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6821","nonce":3467856022}]},"public_addr":"172.21.15.82:6817/3467856022","cluster_addr":"172.21.15.82:6819/3467856022","heartbeat_back_addr":"172.21.15.82:6823/3467856022","heartbeat_front_addr":"172.21.15.82:6821/3467856022","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6801","nonce":968508548}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6806","nonce":968508548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6814","nonce":968508548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6810","nonce":968508548}]},"public_addr":"172.21.15.82:6801/968508548","cluster_addr":"172.21.15.82:6806/968508548","heartbeat_back_addr":"172.21.15.82:6814/968508548","heartbeat_front_addr":"172.21.15.82:6810/968508548","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":52,"up_from":59,"up_thru":95,"down_at":53,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":1490604388},{"type":"v1","addr":"172.21.15.82:6825","nonce":1490604388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":1490604388},{"type":"v1","addr":"172.21.15.82:6827","nonce":1490604388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":1490604388},{"type":"v1","addr":"172.21.15.82:6831","nonce":1490604388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":1490604388},{"type":"v1","addr":"172.21.15.82:6829","nonce":1490604388}]},"public_addr":"172.21.15.82:6825/1490604388","cluster_addr":"172.21.15.82:6827/1490604388","heartbeat_back_addr":"172.21.15.82:6831/1490604388","heartbeat_front_addr":"172.21.15.82:6829/1490604388","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:57:51.786809+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:59:08.569659+0000","dead_epoch":78},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:58:01.545062+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:58:13.383508+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:58:24.159372+0000","dead_epoch":26},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:58:11.960109+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:58:10.360171+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:58:06.145865+0000","dead_epoch":53}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T12:59:15.098 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T12:59:15.098 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T12:59:15.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:15.207+0000 7fee58523640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:15.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:15.223+0000 7fee58523640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:15.400 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T12:59:15.419 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T12:59:15.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:15.531+0000 7f202ba85640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:15.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:15.547+0000 7f202ba85640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:15.727 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:59:15.727 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T12:59:15.744 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T12:59:15.745 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T12:59:15.745 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T12:59:15.745 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T12:59:15.745 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 3, 5, 6, 4, 7, 1] 2024-03-30T12:59:15.745 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T12:59:15.745 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:59:15.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:15.847+0000 7fd2c9dc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:15.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:15.859+0000 7fd2c9dc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:16.032 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:59:16.032 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T12:59:16.048 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":205,"stamp":"2024-03-30T12:59:15.845135+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":55109427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":49,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":319492,"kb_used_data":101760,"kb_used_omap":20,"kb_used_meta":217579,"kb_avail":754655228,"statfs":{"total":773094113280,"available":772766953472,"internally_reserved":0,"allocated":104202240,"data_stored":99025891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21164,"internal_metadata":222801236},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001045"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":208,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-30T12:59:14.119761+0000","last_change":"2024-03-30T12:58:40.339390+0000","last_active":"2024-03-30T12:59:14.119761+0000","last_peered":"2024-03-30T12:59:14.119761+0000","last_clean":"2024-03-30T12:59:14.119761+0000","last_became_active":"2024-03-30T12:57:16.931067+0000","last_became_peered":"2024-03-30T12:57:16.931067+0000","last_unstale":"2024-03-30T12:59:14.119761+0000","last_undegraded":"2024-03-30T12:59:14.119761+0000","last_fullsized":"2024-03-30T12:59:14.119761+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:58:40.339096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:40.339096+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:53.541716+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":187,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-30T12:59:10.082184+0000","last_change":"2024-03-30T12:58:30.264910+0000","last_active":"2024-03-30T12:59:10.082184+0000","last_peered":"2024-03-30T12:59:10.082184+0000","last_clean":"2024-03-30T12:59:10.082184+0000","last_became_active":"2024-03-30T12:58:29.543016+0000","last_became_peered":"2024-03-30T12:58:29.543016+0000","last_unstale":"2024-03-30T12:59:10.082184+0000","last_undegraded":"2024-03-30T12:59:10.082184+0000","last_fullsized":"2024-03-30T12:59:10.082184+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T12:58:30.264658+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:50.191800+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":145,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-30T12:59:11.081745+0000","last_change":"2024-03-30T12:58:58.961161+0000","last_active":"2024-03-30T12:59:11.081745+0000","last_peered":"2024-03-30T12:59:11.081745+0000","last_clean":"2024-03-30T12:59:11.081745+0000","last_became_active":"2024-03-30T12:58:58.960224+0000","last_became_peered":"2024-03-30T12:58:58.960224+0000","last_unstale":"2024-03-30T12:59:11.081745+0000","last_undegraded":"2024-03-30T12:59:11.081745+0000","last_fullsized":"2024-03-30T12:59:11.081745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:58:23.156870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:23.156870+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:37.567491+0000","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":80,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-30T12:59:11.081724+0000","last_change":"2024-03-30T12:58:59.978677+0000","last_active":"2024-03-30T12:59:11.081724+0000","last_peered":"2024-03-30T12:59:11.081724+0000","last_clean":"2024-03-30T12:59:11.081724+0000","last_became_active":"2024-03-30T12:58:59.977351+0000","last_became_peered":"2024-03-30T12:58:59.977351+0000","last_unstale":"2024-03-30T12:59:11.081724+0000","last_undegraded":"2024-03-30T12:59:11.081724+0000","last_fullsized":"2024-03-30T12:59:11.081724+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T12:57:51.514144+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:14.677574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":188,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-30T12:59:13.952536+0000","last_change":"2024-03-30T12:58:33.818975+0000","last_active":"2024-03-30T12:59:13.952536+0000","last_peered":"2024-03-30T12:59:13.952536+0000","last_clean":"2024-03-30T12:59:13.952536+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T12:59:13.952536+0000","last_undegraded":"2024-03-30T12:59:13.952536+0000","last_fullsized":"2024-03-30T12:59:13.952536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:58:33.818576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:33.818576+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:00.736977+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":199,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-30T12:59:12.098249+0000","last_change":"2024-03-30T12:58:01.131164+0000","last_active":"2024-03-30T12:59:12.098249+0000","last_peered":"2024-03-30T12:59:12.098249+0000","last_clean":"2024-03-30T12:59:12.098249+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T12:59:12.098249+0000","last_undegraded":"2024-03-30T12:59:12.098249+0000","last_fullsized":"2024-03-30T12:59:12.098249+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:58:01.130774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:01.130774+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:29.177753+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":159,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-30T12:59:11.081702+0000","last_change":"2024-03-30T12:58:13.024975+0000","last_active":"2024-03-30T12:59:11.081702+0000","last_peered":"2024-03-30T12:59:11.081702+0000","last_clean":"2024-03-30T12:59:11.081702+0000","last_became_active":"2024-03-30T12:58:12.369218+0000","last_became_peered":"2024-03-30T12:58:12.369218+0000","last_unstale":"2024-03-30T12:59:11.081702+0000","last_undegraded":"2024-03-30T12:59:11.081702+0000","last_fullsized":"2024-03-30T12:59:11.081702+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T12:58:13.024706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:13.024706+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:36.211866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":190,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-30T12:59:12.097659+0000","last_change":"2024-03-30T12:58:59.114692+0000","last_active":"2024-03-30T12:59:12.097659+0000","last_peered":"2024-03-30T12:59:12.097659+0000","last_clean":"2024-03-30T12:59:12.097659+0000","last_became_active":"2024-03-30T12:58:59.114041+0000","last_became_peered":"2024-03-30T12:58:59.114041+0000","last_unstale":"2024-03-30T12:59:12.097659+0000","last_undegraded":"2024-03-30T12:59:12.097659+0000","last_fullsized":"2024-03-30T12:59:12.097659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:58:23.190883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:23.190883+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:49.009057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7344560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":217,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-30T12:59:12.098261+0000","last_change":"2024-03-30T12:58:11.358195+0000","last_active":"2024-03-30T12:59:12.098261+0000","last_peered":"2024-03-30T12:59:12.098261+0000","last_clean":"2024-03-30T12:59:12.098261+0000","last_became_active":"2024-03-30T12:58:11.357621+0000","last_became_peered":"2024-03-30T12:58:11.357621+0000","last_unstale":"2024-03-30T12:59:12.098261+0000","last_undegraded":"2024-03-30T12:59:12.098261+0000","last_fullsized":"2024-03-30T12:59:12.098261+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:58:04.171994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:04.171994+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:23.627732+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":155,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-30T12:59:10.082192+0000","last_change":"2024-03-30T12:58:31.220524+0000","last_active":"2024-03-30T12:59:10.082192+0000","last_peered":"2024-03-30T12:59:10.082192+0000","last_clean":"2024-03-30T12:59:10.082192+0000","last_became_active":"2024-03-30T12:57:16.930714+0000","last_became_peered":"2024-03-30T12:57:16.930714+0000","last_unstale":"2024-03-30T12:59:10.082192+0000","last_undegraded":"2024-03-30T12:59:10.082192+0000","last_fullsized":"2024-03-30T12:59:10.082192+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:58:31.220288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:31.220288+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:41.159203+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":229,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-30T12:59:14.120016+0000","last_change":"2024-03-30T12:58:19.389708+0000","last_active":"2024-03-30T12:59:14.120016+0000","last_peered":"2024-03-30T12:59:14.120016+0000","last_clean":"2024-03-30T12:59:14.120016+0000","last_became_active":"2024-03-30T12:57:16.930513+0000","last_became_peered":"2024-03-30T12:57:16.930513+0000","last_unstale":"2024-03-30T12:59:14.120016+0000","last_undegraded":"2024-03-30T12:59:14.120016+0000","last_fullsized":"2024-03-30T12:59:14.120016+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:58:19.389431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:19.389431+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:31.361294+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":214,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-30T12:59:14.119862+0000","last_change":"2024-03-30T12:58:23.382393+0000","last_active":"2024-03-30T12:59:14.119862+0000","last_peered":"2024-03-30T12:59:14.119862+0000","last_clean":"2024-03-30T12:59:14.119862+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T12:59:14.119862+0000","last_undegraded":"2024-03-30T12:59:14.119862+0000","last_fullsized":"2024-03-30T12:59:14.119862+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T12:58:23.382129+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T12:58:23.382129+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:23.404087+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":200,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-30T12:59:13.952530+0000","last_change":"2024-03-30T12:58:14.711329+0000","last_active":"2024-03-30T12:59:13.952530+0000","last_peered":"2024-03-30T12:59:13.952530+0000","last_clean":"2024-03-30T12:59:13.952530+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T12:59:13.952530+0000","last_undegraded":"2024-03-30T12:59:13.952530+0000","last_fullsized":"2024-03-30T12:59:13.952530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:58:14.710869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:14.710869+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:25.694672+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":117,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-30T12:59:11.088606+0000","last_change":"2024-03-30T12:59:11.088606+0000","last_active":"2024-03-30T12:59:11.088606+0000","last_peered":"2024-03-30T12:59:11.088606+0000","last_clean":"2024-03-30T12:59:11.088606+0000","last_became_active":"2024-03-30T12:59:11.087652+0000","last_became_peered":"2024-03-30T12:59:11.087652+0000","last_unstale":"2024-03-30T12:59:11.088606+0000","last_undegraded":"2024-03-30T12:59:11.088606+0000","last_fullsized":"2024-03-30T12:59:11.088606+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:56:56.870694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:56.870694+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":208,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-30T12:59:13.956389+0000","last_change":"2024-03-30T12:59:13.956465+0000","last_active":"2024-03-30T12:59:13.956389+0000","last_peered":"2024-03-30T12:59:13.956389+0000","last_clean":"2024-03-30T12:59:13.956389+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T12:59:13.956389+0000","last_undegraded":"2024-03-30T12:59:13.956389+0000","last_fullsized":"2024-03-30T12:59:13.956389+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:59:13.956186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:13.956186+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:35.452148+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":180,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-30T12:59:12.097679+0000","last_change":"2024-03-30T12:58:59.113868+0000","last_active":"2024-03-30T12:59:12.097679+0000","last_peered":"2024-03-30T12:59:12.097679+0000","last_clean":"2024-03-30T12:59:12.097679+0000","last_became_active":"2024-03-30T12:58:59.113215+0000","last_became_peered":"2024-03-30T12:58:59.113215+0000","last_unstale":"2024-03-30T12:59:12.097679+0000","last_undegraded":"2024-03-30T12:59:12.097679+0000","last_fullsized":"2024-03-30T12:59:12.097679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:58:34.875943+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:58:34.875943+0000","last_clean_scrub_stamp":"2024-03-30T12:58:34.875943+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:01.009498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":55109427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":95059968,"data_stored":95059968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509487,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43276,"kb_used_data":16248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94328564,"statfs":{"total":96636764160,"available":96592449536,"internally_reserved":0,"allocated":16637952,"data_stored":15977312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542191,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46776,"kb_used_data":19748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94325064,"statfs":{"total":96636764160,"available":96588865536,"internally_reserved":0,"allocated":20221952,"data_stored":19574914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822680,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40168,"kb_used_data":12636,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94331672,"statfs":{"total":96636764160,"available":96595632128,"internally_reserved":0,"allocated":12939264,"data_stored":12312592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4399,"internal_metadata":28176081},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509488,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34472,"kb_used_data":7452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337368,"statfs":{"total":96636764160,"available":96601464832,"internally_reserved":0,"allocated":7630848,"data_stored":6977009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285573,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33816,"kb_used_data":6284,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94338024,"statfs":{"total":96636764160,"available":96602136576,"internally_reserved":0,"allocated":6434816,"data_stored":5787778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":28176082},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509488,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41664,"kb_used_data":14640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330176,"statfs":{"total":96636764160,"available":96594100224,"internally_reserved":0,"allocated":14991360,"data_stored":14330720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":59,"seq":253403070479,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38676,"kb_used_data":11136,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94333164,"statfs":{"total":96636764160,"available":96597159936,"internally_reserved":0,"allocated":11403264,"data_stored":10769820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4391,"internal_metadata":28176089},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574896,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40644,"kb_used_data":13616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331196,"statfs":{"total":96636764160,"available":96595144704,"internally_reserved":0,"allocated":13942784,"data_stored":13295746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19079168,"data_stored":19079168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:59:16.049 INFO:tasks.thrashosds.thrasher:Choosing PG 1.2 with acting set [7, 0, 4] 2024-03-30T12:59:16.050 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 2, 3, 5, 6, 4, 7, 1] 2024-03-30T12:59:16.050 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-30T12:59:16.050 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T12:59:16.119 INFO:tasks.ceph.osd.7:Stopped 2024-03-30T12:59:16.119 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 2, 3, 5, 6, 4, 7, 1] 2024-03-30T12:59:16.120 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 7 2024-03-30T12:59:16.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:16.235+0000 7f645927d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:16.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:16.251+0000 7f645927d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:16.437 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T12:59:16.431+0000 7fbb38f08640 -1 mon.a@0(leader).osd e101 definitely_dead 0 2024-03-30T12:59:17.151 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.7. 2024-03-30T12:59:17.179 INFO:tasks.thrashosds.thrasher:dead_osds=[7], live_osds=[0, 2, 3, 5, 6, 4, 1] 2024-03-30T12:59:17.179 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T12:59:17.179 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:59:17.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:17.287+0000 7feeeb8d2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:17.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:17.299+0000 7feeeb8d2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:17.478 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:59:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T12:59:17.497 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2024-03-30T12:59:17.147734+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":55109427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":43,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":280876,"kb_used_data":90684,"kb_used_omap":16,"kb_used_meta":190063,"kb_avail":660322004,"statfs":{"total":676457349120,"available":676169732096,"internally_reserved":0,"allocated":92860416,"data_stored":88283543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16773,"internal_metadata":194625147},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.042264"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":208,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-30T12:59:14.119761+0000","last_change":"2024-03-30T12:58:40.339390+0000","last_active":"2024-03-30T12:59:14.119761+0000","last_peered":"2024-03-30T12:59:14.119761+0000","last_clean":"2024-03-30T12:59:14.119761+0000","last_became_active":"2024-03-30T12:57:16.931067+0000","last_became_peered":"2024-03-30T12:57:16.931067+0000","last_unstale":"2024-03-30T12:59:14.119761+0000","last_undegraded":"2024-03-30T12:59:14.119761+0000","last_fullsized":"2024-03-30T12:59:14.119761+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:58:40.339096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:40.339096+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:53.541716+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":197,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-30T12:59:15.119921+0000","last_change":"2024-03-30T12:58:30.264910+0000","last_active":"2024-03-30T12:59:15.119921+0000","last_peered":"2024-03-30T12:59:15.119921+0000","last_clean":"2024-03-30T12:59:15.119921+0000","last_became_active":"2024-03-30T12:58:29.543016+0000","last_became_peered":"2024-03-30T12:58:29.543016+0000","last_unstale":"2024-03-30T12:59:15.119921+0000","last_undegraded":"2024-03-30T12:59:15.119921+0000","last_fullsized":"2024-03-30T12:59:15.119921+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T12:58:30.264658+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:50.191800+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":145,"reported_epoch":96,"state":"stale+active+clean","last_fresh":"2024-03-30T12:59:11.081745+0000","last_change":"2024-03-30T12:58:58.961161+0000","last_active":"2024-03-30T12:59:11.081745+0000","last_peered":"2024-03-30T12:59:11.081745+0000","last_clean":"2024-03-30T12:59:11.081745+0000","last_became_active":"2024-03-30T12:58:58.960224+0000","last_became_peered":"2024-03-30T12:58:58.960224+0000","last_unstale":"2024-03-30T12:59:16.136925+0000","last_undegraded":"2024-03-30T12:59:11.081745+0000","last_fullsized":"2024-03-30T12:59:11.081745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:58:23.156870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:23.156870+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:37.567491+0000","scrub_duration":231,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":80,"reported_epoch":96,"state":"stale+active+clean","last_fresh":"2024-03-30T12:59:11.081724+0000","last_change":"2024-03-30T12:58:59.978677+0000","last_active":"2024-03-30T12:59:11.081724+0000","last_peered":"2024-03-30T12:59:11.081724+0000","last_clean":"2024-03-30T12:59:11.081724+0000","last_became_active":"2024-03-30T12:58:59.977351+0000","last_became_peered":"2024-03-30T12:58:59.977351+0000","last_unstale":"2024-03-30T12:59:16.136930+0000","last_undegraded":"2024-03-30T12:59:11.081724+0000","last_fullsized":"2024-03-30T12:59:11.081724+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T12:57:51.514144+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:14.677574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":188,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-30T12:59:13.952536+0000","last_change":"2024-03-30T12:58:33.818975+0000","last_active":"2024-03-30T12:59:13.952536+0000","last_peered":"2024-03-30T12:59:13.952536+0000","last_clean":"2024-03-30T12:59:13.952536+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T12:59:13.952536+0000","last_undegraded":"2024-03-30T12:59:13.952536+0000","last_fullsized":"2024-03-30T12:59:13.952536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:58:33.818576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:33.818576+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:00.736977+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":199,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-30T12:59:12.098249+0000","last_change":"2024-03-30T12:58:01.131164+0000","last_active":"2024-03-30T12:59:12.098249+0000","last_peered":"2024-03-30T12:59:12.098249+0000","last_clean":"2024-03-30T12:59:12.098249+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T12:59:12.098249+0000","last_undegraded":"2024-03-30T12:59:12.098249+0000","last_fullsized":"2024-03-30T12:59:12.098249+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:58:01.130774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:01.130774+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:29.177753+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":159,"reported_epoch":96,"state":"stale+active+clean","last_fresh":"2024-03-30T12:59:11.081702+0000","last_change":"2024-03-30T12:58:13.024975+0000","last_active":"2024-03-30T12:59:11.081702+0000","last_peered":"2024-03-30T12:59:11.081702+0000","last_clean":"2024-03-30T12:59:11.081702+0000","last_became_active":"2024-03-30T12:58:12.369218+0000","last_became_peered":"2024-03-30T12:58:12.369218+0000","last_unstale":"2024-03-30T12:59:16.136935+0000","last_undegraded":"2024-03-30T12:59:11.081702+0000","last_fullsized":"2024-03-30T12:59:11.081702+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T12:58:13.024706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:13.024706+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:36.211866+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":190,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-30T12:59:12.097659+0000","last_change":"2024-03-30T12:58:59.114692+0000","last_active":"2024-03-30T12:59:12.097659+0000","last_peered":"2024-03-30T12:59:12.097659+0000","last_clean":"2024-03-30T12:59:12.097659+0000","last_became_active":"2024-03-30T12:58:59.114041+0000","last_became_peered":"2024-03-30T12:58:59.114041+0000","last_unstale":"2024-03-30T12:59:12.097659+0000","last_undegraded":"2024-03-30T12:59:12.097659+0000","last_fullsized":"2024-03-30T12:59:12.097659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:58:23.190883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:23.190883+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:49.009057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7344560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":217,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-30T12:59:12.098261+0000","last_change":"2024-03-30T12:58:11.358195+0000","last_active":"2024-03-30T12:59:12.098261+0000","last_peered":"2024-03-30T12:59:12.098261+0000","last_clean":"2024-03-30T12:59:12.098261+0000","last_became_active":"2024-03-30T12:58:11.357621+0000","last_became_peered":"2024-03-30T12:58:11.357621+0000","last_unstale":"2024-03-30T12:59:12.098261+0000","last_undegraded":"2024-03-30T12:59:12.098261+0000","last_fullsized":"2024-03-30T12:59:12.098261+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:58:04.171994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:04.171994+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:23.627732+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":165,"reported_epoch":100,"state":"active+clean","last_fresh":"2024-03-30T12:59:15.119806+0000","last_change":"2024-03-30T12:58:31.220524+0000","last_active":"2024-03-30T12:59:15.119806+0000","last_peered":"2024-03-30T12:59:15.119806+0000","last_clean":"2024-03-30T12:59:15.119806+0000","last_became_active":"2024-03-30T12:57:16.930714+0000","last_became_peered":"2024-03-30T12:57:16.930714+0000","last_unstale":"2024-03-30T12:59:15.119806+0000","last_undegraded":"2024-03-30T12:59:15.119806+0000","last_fullsized":"2024-03-30T12:59:15.119806+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:58:31.220288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:31.220288+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:41.159203+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":229,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-30T12:59:14.120016+0000","last_change":"2024-03-30T12:58:19.389708+0000","last_active":"2024-03-30T12:59:14.120016+0000","last_peered":"2024-03-30T12:59:14.120016+0000","last_clean":"2024-03-30T12:59:14.120016+0000","last_became_active":"2024-03-30T12:57:16.930513+0000","last_became_peered":"2024-03-30T12:57:16.930513+0000","last_unstale":"2024-03-30T12:59:14.120016+0000","last_undegraded":"2024-03-30T12:59:14.120016+0000","last_fullsized":"2024-03-30T12:59:14.120016+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:58:19.389431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:19.389431+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:31.361294+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":214,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-03-30T12:59:14.119862+0000","last_change":"2024-03-30T12:58:23.382393+0000","last_active":"2024-03-30T12:59:14.119862+0000","last_peered":"2024-03-30T12:59:14.119862+0000","last_clean":"2024-03-30T12:59:14.119862+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T12:59:14.119862+0000","last_undegraded":"2024-03-30T12:59:14.119862+0000","last_fullsized":"2024-03-30T12:59:14.119862+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T12:58:23.382129+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T12:58:23.382129+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:23.404087+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":200,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-30T12:59:13.952530+0000","last_change":"2024-03-30T12:58:14.711329+0000","last_active":"2024-03-30T12:59:13.952530+0000","last_peered":"2024-03-30T12:59:13.952530+0000","last_clean":"2024-03-30T12:59:13.952530+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T12:59:13.952530+0000","last_undegraded":"2024-03-30T12:59:13.952530+0000","last_fullsized":"2024-03-30T12:59:13.952530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:58:14.710869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:14.710869+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:25.694672+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":117,"reported_epoch":96,"state":"stale+active+clean","last_fresh":"2024-03-30T12:59:11.088606+0000","last_change":"2024-03-30T12:59:11.088606+0000","last_active":"2024-03-30T12:59:11.088606+0000","last_peered":"2024-03-30T12:59:11.088606+0000","last_clean":"2024-03-30T12:59:11.088606+0000","last_became_active":"2024-03-30T12:59:11.087652+0000","last_became_peered":"2024-03-30T12:59:11.087652+0000","last_unstale":"2024-03-30T12:59:16.136939+0000","last_undegraded":"2024-03-30T12:59:11.088606+0000","last_fullsized":"2024-03-30T12:59:11.088606+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:56:56.870694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:56:56.870694+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":208,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-03-30T12:59:13.956389+0000","last_change":"2024-03-30T12:59:13.956465+0000","last_active":"2024-03-30T12:59:13.956389+0000","last_peered":"2024-03-30T12:59:13.956389+0000","last_clean":"2024-03-30T12:59:13.956389+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T12:59:13.956389+0000","last_undegraded":"2024-03-30T12:59:13.956389+0000","last_fullsized":"2024-03-30T12:59:13.956389+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:59:13.956186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:13.956186+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:35.452148+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":180,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-30T12:59:12.097679+0000","last_change":"2024-03-30T12:58:59.113868+0000","last_active":"2024-03-30T12:59:12.097679+0000","last_peered":"2024-03-30T12:59:12.097679+0000","last_clean":"2024-03-30T12:59:12.097679+0000","last_became_active":"2024-03-30T12:58:59.113215+0000","last_became_peered":"2024-03-30T12:58:59.113215+0000","last_unstale":"2024-03-30T12:59:12.097679+0000","last_undegraded":"2024-03-30T12:59:12.097679+0000","last_fullsized":"2024-03-30T12:59:12.097679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:58:34.875943+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:58:34.875943+0000","last_clean_scrub_stamp":"2024-03-30T12:58:34.875943+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:01.009498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":55109427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":95059968,"data_stored":95059968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509487,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43276,"kb_used_data":16248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94328564,"statfs":{"total":96636764160,"available":96592449536,"internally_reserved":0,"allocated":16637952,"data_stored":15977312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542191,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46776,"kb_used_data":19748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94325064,"statfs":{"total":96636764160,"available":96588865536,"internally_reserved":0,"allocated":20221952,"data_stored":19574914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822681,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40228,"kb_used_data":12696,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94331612,"statfs":{"total":96636764160,"available":96595570688,"internally_reserved":0,"allocated":13000704,"data_stored":12340064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4399,"internal_metadata":28176081},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509488,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34472,"kb_used_data":7452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337368,"statfs":{"total":96636764160,"available":96601464832,"internally_reserved":0,"allocated":7630848,"data_stored":6977009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285573,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33816,"kb_used_data":6284,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94338024,"statfs":{"total":96636764160,"available":96602136576,"internally_reserved":0,"allocated":6434816,"data_stored":5787778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":28176082},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509488,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41664,"kb_used_data":14640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330176,"statfs":{"total":96636764160,"available":96594100224,"internally_reserved":0,"allocated":14991360,"data_stored":14330720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574896,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40644,"kb_used_data":13616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331196,"statfs":{"total":96636764160,"available":96595144704,"internally_reserved":0,"allocated":13942784,"data_stored":13295746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19079168,"data_stored":19079168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:59:17.498 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T12:59:42.499 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T12:59:42.500 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T12:59:42.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:42.622+0000 7f8ed2a33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:42.634+0000 7f8ed2a33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:42.823 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T12:59:42.823 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T12:59:42.841 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-03-30T12:59:41.853583+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":84554895,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":295528,"kb_used_data":101232,"kb_used_omap":16,"kb_used_meta":194223,"kb_avail":660307352,"statfs":{"total":676457349120,"available":676154728448,"internally_reserved":0,"allocated":103661568,"data_stored":98802665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16773,"internal_metadata":198884987},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003844"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":217,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-30T12:59:20.525434+0000","last_change":"2024-03-30T12:58:40.339390+0000","last_active":"2024-03-30T12:59:20.525434+0000","last_peered":"2024-03-30T12:59:20.525434+0000","last_clean":"2024-03-30T12:59:20.525434+0000","last_became_active":"2024-03-30T12:57:16.931067+0000","last_became_peered":"2024-03-30T12:57:16.931067+0000","last_unstale":"2024-03-30T12:59:20.525434+0000","last_undegraded":"2024-03-30T12:59:20.525434+0000","last_fullsized":"2024-03-30T12:59:20.525434+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:58:40.339096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:40.339096+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:53.541716+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":215,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-30T12:59:20.165012+0000","last_change":"2024-03-30T12:59:18.198290+0000","last_active":"2024-03-30T12:59:20.165012+0000","last_peered":"2024-03-30T12:59:20.165012+0000","last_clean":"2024-03-30T12:59:20.165012+0000","last_became_active":"2024-03-30T12:59:18.166875+0000","last_became_peered":"2024-03-30T12:59:18.166875+0000","last_unstale":"2024-03-30T12:59:20.165012+0000","last_undegraded":"2024-03-30T12:59:20.165012+0000","last_fullsized":"2024-03-30T12:59:20.165012+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T12:58:30.264658+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:48.823596+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10621892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":21,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-30T12:59:20.165689+0000","last_change":"2024-03-30T12:59:18.250979+0000","last_active":"2024-03-30T12:59:20.165689+0000","last_peered":"2024-03-30T12:59:20.165689+0000","last_clean":"2024-03-30T12:59:20.165689+0000","last_became_active":"2024-03-30T12:59:18.164875+0000","last_became_peered":"2024-03-30T12:59:18.164875+0000","last_unstale":"2024-03-30T12:59:20.165689+0000","last_undegraded":"2024-03-30T12:59:20.165689+0000","last_fullsized":"2024-03-30T12:59:20.165689+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:58:23.156870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:23.156870+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:45.690394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":27,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-30T12:59:20.898082+0000","last_change":"2024-03-30T12:59:20.898167+0000","last_active":"2024-03-30T12:59:20.898082+0000","last_peered":"2024-03-30T12:59:20.898082+0000","last_clean":"2024-03-30T12:59:20.898082+0000","last_became_active":"2024-03-30T12:59:18.159990+0000","last_became_peered":"2024-03-30T12:59:18.159990+0000","last_unstale":"2024-03-30T12:59:20.898082+0000","last_undegraded":"2024-03-30T12:59:20.898082+0000","last_fullsized":"2024-03-30T12:59:20.898082+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:59:20.897870+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T12:59:20.897870+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:30.595229+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":200,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-30T12:59:21.010624+0000","last_change":"2024-03-30T12:58:33.818975+0000","last_active":"2024-03-30T12:59:21.010624+0000","last_peered":"2024-03-30T12:59:21.010624+0000","last_clean":"2024-03-30T12:59:21.010624+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T12:59:21.010624+0000","last_undegraded":"2024-03-30T12:59:21.010624+0000","last_fullsized":"2024-03-30T12:59:21.010624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:58:33.818576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:33.818576+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:00.736977+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":221,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-30T12:59:32.471388+0000","last_change":"2024-03-30T12:59:32.471464+0000","last_active":"2024-03-30T12:59:32.471388+0000","last_peered":"2024-03-30T12:59:32.471388+0000","last_clean":"2024-03-30T12:59:32.471388+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T12:59:32.471388+0000","last_undegraded":"2024-03-30T12:59:32.471388+0000","last_fullsized":"2024-03-30T12:59:32.471388+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:59:32.471198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:32.471198+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:45.757823+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":42,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-30T12:59:20.919631+0000","last_change":"2024-03-30T12:59:20.919631+0000","last_active":"2024-03-30T12:59:20.919631+0000","last_peered":"2024-03-30T12:59:20.919631+0000","last_clean":"2024-03-30T12:59:20.919631+0000","last_became_active":"2024-03-30T12:59:18.171242+0000","last_became_peered":"2024-03-30T12:59:18.171242+0000","last_unstale":"2024-03-30T12:59:20.919631+0000","last_undegraded":"2024-03-30T12:59:20.919631+0000","last_fullsized":"2024-03-30T12:59:20.919631+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T12:59:20.919233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:20.919233+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:34.567410+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":204,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-30T12:59:20.164463+0000","last_change":"2024-03-30T12:58:59.114692+0000","last_active":"2024-03-30T12:59:20.164463+0000","last_peered":"2024-03-30T12:59:20.164463+0000","last_clean":"2024-03-30T12:59:20.164463+0000","last_became_active":"2024-03-30T12:58:59.114041+0000","last_became_peered":"2024-03-30T12:58:59.114041+0000","last_unstale":"2024-03-30T12:59:20.164463+0000","last_undegraded":"2024-03-30T12:59:20.164463+0000","last_fullsized":"2024-03-30T12:59:20.164463+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:58:23.190883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:23.190883+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:49.009057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7344560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":250,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-30T12:59:20.509718+0000","last_change":"2024-03-30T12:59:18.470688+0000","last_active":"2024-03-30T12:59:20.509718+0000","last_peered":"2024-03-30T12:59:20.509718+0000","last_clean":"2024-03-30T12:59:20.509718+0000","last_became_active":"2024-03-30T12:59:18.167961+0000","last_became_peered":"2024-03-30T12:59:18.167961+0000","last_unstale":"2024-03-30T12:59:20.509718+0000","last_undegraded":"2024-03-30T12:59:20.509718+0000","last_fullsized":"2024-03-30T12:59:20.509718+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:59:18.470423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:18.470423+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:19.720378+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15226944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":174,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-30T12:59:20.164922+0000","last_change":"2024-03-30T12:58:31.220524+0000","last_active":"2024-03-30T12:59:20.164922+0000","last_peered":"2024-03-30T12:59:20.164922+0000","last_clean":"2024-03-30T12:59:20.164922+0000","last_became_active":"2024-03-30T12:57:16.930714+0000","last_became_peered":"2024-03-30T12:57:16.930714+0000","last_unstale":"2024-03-30T12:59:20.164922+0000","last_undegraded":"2024-03-30T12:59:20.164922+0000","last_fullsized":"2024-03-30T12:59:20.164922+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:58:31.220288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:58:31.220288+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T12:59:41.159203+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":246,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-30T12:59:37.603668+0000","last_change":"2024-03-30T12:59:37.603807+0000","last_active":"2024-03-30T12:59:37.603668+0000","last_peered":"2024-03-30T12:59:37.603668+0000","last_clean":"2024-03-30T12:59:37.603668+0000","last_became_active":"2024-03-30T12:57:16.930513+0000","last_became_peered":"2024-03-30T12:57:16.930513+0000","last_unstale":"2024-03-30T12:59:37.603668+0000","last_undegraded":"2024-03-30T12:59:37.603668+0000","last_fullsized":"2024-03-30T12:59:37.603668+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:59:37.603454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:37.603454+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:56.515014+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":231,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-30T12:59:28.587371+0000","last_change":"2024-03-30T12:59:28.587515+0000","last_active":"2024-03-30T12:59:28.587371+0000","last_peered":"2024-03-30T12:59:28.587371+0000","last_clean":"2024-03-30T12:59:28.587371+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T12:59:28.587371+0000","last_undegraded":"2024-03-30T12:59:28.587371+0000","last_fullsized":"2024-03-30T12:59:28.587371+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T12:59:28.587085+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T12:59:28.587085+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:52.894911+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":220,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-30T12:59:26.888948+0000","last_change":"2024-03-30T12:59:26.889088+0000","last_active":"2024-03-30T12:59:26.888948+0000","last_peered":"2024-03-30T12:59:26.888948+0000","last_clean":"2024-03-30T12:59:26.888948+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T12:59:26.888948+0000","last_undegraded":"2024-03-30T12:59:26.888948+0000","last_fullsized":"2024-03-30T12:59:26.888948+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:59:26.888729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:26.888729+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:49.934982+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":25,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-30T12:59:24.867607+0000","last_change":"2024-03-30T12:59:24.867674+0000","last_active":"2024-03-30T12:59:24.867607+0000","last_peered":"2024-03-30T12:59:24.867607+0000","last_clean":"2024-03-30T12:59:24.867607+0000","last_became_active":"2024-03-30T12:59:18.168404+0000","last_became_peered":"2024-03-30T12:59:18.168404+0000","last_unstale":"2024-03-30T12:59:24.867607+0000","last_undegraded":"2024-03-30T12:59:24.867607+0000","last_fullsized":"2024-03-30T12:59:24.867607+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T12:59:24.867416+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:34.132102+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":220,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-30T12:59:21.010652+0000","last_change":"2024-03-30T12:59:13.956465+0000","last_active":"2024-03-30T12:59:21.010652+0000","last_peered":"2024-03-30T12:59:21.010652+0000","last_clean":"2024-03-30T12:59:21.010652+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T12:59:21.010652+0000","last_undegraded":"2024-03-30T12:59:21.010652+0000","last_fullsized":"2024-03-30T12:59:21.010652+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:59:13.956186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:13.956186+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:35.452148+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":194,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-30T12:59:20.164799+0000","last_change":"2024-03-30T12:58:59.113868+0000","last_active":"2024-03-30T12:59:20.164799+0000","last_peered":"2024-03-30T12:59:20.164799+0000","last_clean":"2024-03-30T12:59:20.164799+0000","last_became_active":"2024-03-30T12:58:59.113215+0000","last_became_peered":"2024-03-30T12:58:59.113215+0000","last_unstale":"2024-03-30T12:59:20.164799+0000","last_undegraded":"2024-03-30T12:59:20.164799+0000","last_fullsized":"2024-03-30T12:59:20.164799+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:58:34.875943+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:58:34.875943+0000","last_clean_scrub_stamp":"2024-03-30T12:58:34.875943+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:01.009498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":84554895,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":105345024,"data_stored":105345024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509492,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43324,"kb_used_data":16308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94328516,"statfs":{"total":96636764160,"available":96592400384,"internally_reserved":0,"allocated":16699392,"data_stored":16005263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542196,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47488,"kb_used_data":20472,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94324352,"statfs":{"total":96636764160,"available":96588136448,"internally_reserved":0,"allocated":20963328,"data_stored":20269199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822686,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44444,"kb_used_data":12756,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94327396,"statfs":{"total":96636764160,"available":96591253504,"internally_reserved":0,"allocated":13062144,"data_stored":12368015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4399,"internal_metadata":32435921},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509493,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40636,"kb_used_data":13620,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331204,"statfs":{"total":96636764160,"available":96595152896,"internally_reserved":0,"allocated":13946880,"data_stored":13252751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285578,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37204,"kb_used_data":9676,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94334636,"statfs":{"total":96636764160,"available":96598667264,"internally_reserved":0,"allocated":9908224,"data_stored":9214095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":28176082},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509493,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41716,"kb_used_data":14700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330124,"statfs":{"total":96636764160,"available":96594046976,"internally_reserved":0,"allocated":15052800,"data_stored":14358671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574901,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40716,"kb_used_data":13700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331124,"statfs":{"total":96636764160,"available":96595070976,"internally_reserved":0,"allocated":14028800,"data_stored":13334671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8679424,"data_stored":8679424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12718080,"data_stored":12718080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19734528,"data_stored":19734528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T12:59:42.842 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T12:59:42.842 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T12:59:42.843 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2024-03-30T12:59:42.843 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-30T12:59:42.843 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-30T12:59:42.845 INFO:tasks.ceph.osd.7:Started 2024-03-30T12:59:42.846 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-30T12:59:42.911 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:59:42.909+0000 7f89cd5257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:42.919 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:59:42.917+0000 7f89cd5257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:42.919 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:59:42.917+0000 7f89cd5257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:42.963 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T12:59:42.963 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T12:59:42.963 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T12:59:42.963 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T12:59:42.963 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T12:59:42.963 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T12:59:42.963 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-03-30T12:59:42.963 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-03-30T12:59:42.963 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-03-30T12:59:42.964 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-03-30T12:59:42.964 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-03-30T12:59:42.964 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T12:59:42.965 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T12:59:42.965 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-03-30T12:59:43.453 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:59:43.449+0000 7f89cd5257c0 -1 Falling back to public interface 2024-03-30T12:59:47.177 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:59:47.173+0000 7f89cd5257c0 -1 osd.7 100 log_to_monitors true 2024-03-30T12:59:47.966 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-30T12:59:48.088 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T12:59:48.088 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T12:59:48.088 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T12:59:48.088 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T12:59:48.088 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T12:59:48.088 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T12:59:48.088 INFO:teuthology.orchestra.run.smithi082.stderr:bluefs debug_inject_read_zeros 2024-03-30T12:59:48.088 INFO:teuthology.orchestra.run.smithi082.stderr:bluefs files list 2024-03-30T12:59:48.089 INFO:teuthology.orchestra.run.smithi082.stderr:bluefs stats 2024-03-30T12:59:48.089 INFO:teuthology.orchestra.run.smithi082.stderr:bluestore allocator dump block 2024-03-30T12:59:48.089 INFO:teuthology.orchestra.run.smithi082.stderr:bluestore allocator fragmentation block 2024-03-30T12:59:48.089 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T12:59:48.090 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T12:59:48.091 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-03-30T12:59:48.223 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T12:59:48.221+0000 7f89c12f2640 -1 osd.7 100 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T12:59:53.092 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-30T12:59:53.222 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T12:59:53.222 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T12:59:53.222 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T12:59:53.222 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T12:59:53.235 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T12:59:53.235 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-03-30T12:59:53.235 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 7 2024-03-30T12:59:53.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:53.362+0000 7f1bc6d11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:53.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:53.378+0000 7f1bc6d11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:53.559 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T12:59:53.558+0000 7fbb38f08640 -1 mon.a@0(leader).osd e108 definitely_dead 0 2024-03-30T12:59:54.283 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.7. 2024-03-30T12:59:54.316 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-03-30T12:59:54.317 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T12:59:54.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:54.438+0000 7f08ad34e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:54.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:54.454+0000 7f08ad34e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:55.291 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T12:59:55.319 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T12:59:55.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:55.442+0000 7fedfd862640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:55.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:55.458+0000 7fedfd862640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:56.306 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T12:59:56.329 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T12:59:56.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:56.450+0000 7ffa5d54c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:56.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:56.466+0000 7ffa5d54c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:57.308 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T12:59:57.336 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T12:59:57.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:57.454+0000 7fd38186b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:57.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:57.470+0000 7fd38186b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:58.324 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T12:59:58.348 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T12:59:58.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:58.466+0000 7fe6e55ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:58.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:58.486+0000 7fe6e55ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:59.334 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T12:59:59.357 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T12:59:59.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:59.474+0000 7f4959474640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T12:59:59.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T12:59:59.498+0000 7f4959474640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:00.347 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:00:00.373 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:00:00.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:00.490+0000 7fa48795a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:00.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:00.506+0000 7fa48795a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:01.363 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:00:01.389 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:00:01.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:01.506+0000 7effc8760640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:01.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:01.526+0000 7effc8760640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:02.365 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:00:02.392 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:00:02.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:02.510+0000 7f18cb050640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:02.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:02.526+0000 7f18cb050640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:03.382 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:00:03.406 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:00:03.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:03.522+0000 7f338d514640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:03.538+0000 7f338d514640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:04.403 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:00:04.426 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:00:04.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:04.542+0000 7f90bdfa0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:04.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:04.562+0000 7f90bdfa0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:05.408 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:00:05.431 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:00:05.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:05.546+0000 7f6cbb4c6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:05.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:05.562+0000 7f6cbb4c6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:06.418 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:00:06.446 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:00:06.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:06.562+0000 7f0bb1a0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:06.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:06.578+0000 7f0bb1a0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:07.426 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:00:07.453 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:00:07.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:07.570+0000 7fa8669aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:07.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:07.586+0000 7fa8669aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:08.432 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:00:08.459 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:00:08.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:08.578+0000 7eff7936f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:08.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:08.594+0000 7eff7936f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:09.441 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:00:09.466 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:00:09.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:09.582+0000 7fce4f5bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:09.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:09.598+0000 7fce4f5bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:10.466 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:00:10.493 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:00:10.494 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:00:10.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:10.614+0000 7f3c8d7a0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:10.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:10.626+0000 7f3c8d7a0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:10.804 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:00:10.804 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:00:10.822 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-03-30T13:00:10.461498+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":92286016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":330632,"kb_used_data":104136,"kb_used_omap":22,"kb_used_meta":226345,"kb_avail":754644088,"statfs":{"total":773094113280,"available":772755546112,"internally_reserved":0,"allocated":106635264,"data_stored":100129970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22815,"internal_metadata":231778017},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.039870"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":263,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-30T13:00:09.049303+0000","last_change":"2024-03-30T12:59:56.673098+0000","last_active":"2024-03-30T13:00:09.049303+0000","last_peered":"2024-03-30T13:00:09.049303+0000","last_clean":"2024-03-30T13:00:09.049303+0000","last_became_active":"2024-03-30T12:57:16.931067+0000","last_became_peered":"2024-03-30T12:57:16.931067+0000","last_unstale":"2024-03-30T13:00:09.049303+0000","last_undegraded":"2024-03-30T13:00:09.049303+0000","last_fullsized":"2024-03-30T13:00:09.049303+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:59:56.672842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:56.672842+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:15.718248+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":263,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-30T13:00:05.403774+0000","last_change":"2024-03-30T12:59:55.341764+0000","last_active":"2024-03-30T13:00:05.403774+0000","last_peered":"2024-03-30T13:00:05.403774+0000","last_clean":"2024-03-30T13:00:05.403774+0000","last_became_active":"2024-03-30T12:59:55.307854+0000","last_became_peered":"2024-03-30T12:59:55.307854+0000","last_unstale":"2024-03-30T13:00:05.403774+0000","last_undegraded":"2024-03-30T13:00:05.403774+0000","last_fullsized":"2024-03-30T13:00:05.403774+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:59:48.880811+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T12:59:48.880811+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:04.063697+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":13277365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":41,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-30T13:00:07.636309+0000","last_change":"2024-03-30T12:59:55.369991+0000","last_active":"2024-03-30T13:00:07.636309+0000","last_peered":"2024-03-30T13:00:07.636309+0000","last_clean":"2024-03-30T13:00:07.636309+0000","last_became_active":"2024-03-30T12:59:55.313597+0000","last_became_peered":"2024-03-30T12:59:55.313597+0000","last_unstale":"2024-03-30T13:00:07.636309+0000","last_undegraded":"2024-03-30T13:00:07.636309+0000","last_fullsized":"2024-03-30T13:00:07.636309+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:59:47.146312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:47.146312+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:59.164559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":39,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-30T13:00:07.636213+0000","last_change":"2024-03-30T12:59:55.395203+0000","last_active":"2024-03-30T13:00:07.636213+0000","last_peered":"2024-03-30T13:00:07.636213+0000","last_clean":"2024-03-30T13:00:07.636213+0000","last_became_active":"2024-03-30T12:59:55.316713+0000","last_became_peered":"2024-03-30T12:59:55.316713+0000","last_unstale":"2024-03-30T13:00:07.636213+0000","last_undegraded":"2024-03-30T13:00:07.636213+0000","last_fullsized":"2024-03-30T13:00:07.636213+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:59:20.897870+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T12:59:20.897870+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:30.841457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":246,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-30T13:00:08.435905+0000","last_change":"2024-03-30T13:00:01.952302+0000","last_active":"2024-03-30T13:00:08.435905+0000","last_peered":"2024-03-30T13:00:08.435905+0000","last_clean":"2024-03-30T13:00:08.435905+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T13:00:08.435905+0000","last_undegraded":"2024-03-30T13:00:08.435905+0000","last_fullsized":"2024-03-30T13:00:08.435905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:00:01.951942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:01.951942+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:08.852178+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":255,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-30T13:00:07.636104+0000","last_change":"2024-03-30T12:59:32.471464+0000","last_active":"2024-03-30T13:00:07.636104+0000","last_peered":"2024-03-30T13:00:07.636104+0000","last_clean":"2024-03-30T13:00:07.636104+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T13:00:07.636104+0000","last_undegraded":"2024-03-30T13:00:07.636104+0000","last_fullsized":"2024-03-30T13:00:07.636104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:59:32.471198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:32.471198+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:45.757823+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":54,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-30T13:00:07.636218+0000","last_change":"2024-03-30T12:59:55.487031+0000","last_active":"2024-03-30T13:00:07.636218+0000","last_peered":"2024-03-30T13:00:07.636218+0000","last_clean":"2024-03-30T13:00:07.636218+0000","last_became_active":"2024-03-30T12:59:55.317071+0000","last_became_peered":"2024-03-30T12:59:55.317071+0000","last_unstale":"2024-03-30T13:00:07.636218+0000","last_undegraded":"2024-03-30T13:00:07.636218+0000","last_fullsized":"2024-03-30T13:00:07.636218+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T12:59:20.919233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:20.919233+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:21.734360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":248,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-30T13:00:07.424478+0000","last_change":"2024-03-30T12:59:59.675214+0000","last_active":"2024-03-30T13:00:07.424478+0000","last_peered":"2024-03-30T13:00:07.424478+0000","last_clean":"2024-03-30T13:00:07.424478+0000","last_became_active":"2024-03-30T12:58:59.114041+0000","last_became_peered":"2024-03-30T12:58:59.114041+0000","last_unstale":"2024-03-30T13:00:07.424478+0000","last_undegraded":"2024-03-30T13:00:07.424478+0000","last_fullsized":"2024-03-30T13:00:07.424478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:59:59.675050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:59.675050+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:20.917814+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7344560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":297,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-30T13:00:07.636358+0000","last_change":"2024-03-30T12:59:55.347262+0000","last_active":"2024-03-30T13:00:07.636358+0000","last_peered":"2024-03-30T13:00:07.636358+0000","last_clean":"2024-03-30T13:00:07.636358+0000","last_became_active":"2024-03-30T12:59:55.305411+0000","last_became_peered":"2024-03-30T12:59:55.305411+0000","last_unstale":"2024-03-30T13:00:07.636358+0000","last_undegraded":"2024-03-30T13:00:07.636358+0000","last_fullsized":"2024-03-30T13:00:07.636358+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:59:18.470423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:18.470423+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:46.794865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":20302592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":212,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-30T13:00:05.403396+0000","last_change":"2024-03-30T12:59:42.816701+0000","last_active":"2024-03-30T13:00:05.403396+0000","last_peered":"2024-03-30T13:00:05.403396+0000","last_clean":"2024-03-30T13:00:05.403396+0000","last_became_active":"2024-03-30T12:57:16.930714+0000","last_became_peered":"2024-03-30T12:57:16.930714+0000","last_unstale":"2024-03-30T13:00:05.403396+0000","last_undegraded":"2024-03-30T13:00:05.403396+0000","last_fullsized":"2024-03-30T13:00:05.403396+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T12:59:42.816452+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:57.970457+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":285,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-30T13:00:09.049160+0000","last_change":"2024-03-30T12:59:37.603807+0000","last_active":"2024-03-30T13:00:09.049160+0000","last_peered":"2024-03-30T13:00:09.049160+0000","last_clean":"2024-03-30T13:00:09.049160+0000","last_became_active":"2024-03-30T12:57:16.930513+0000","last_became_peered":"2024-03-30T12:57:16.930513+0000","last_unstale":"2024-03-30T13:00:09.049160+0000","last_undegraded":"2024-03-30T13:00:09.049160+0000","last_fullsized":"2024-03-30T13:00:09.049160+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:59:37.603454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:37.603454+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:56.515014+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":270,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-30T13:00:09.049452+0000","last_change":"2024-03-30T12:59:28.587515+0000","last_active":"2024-03-30T13:00:09.049452+0000","last_peered":"2024-03-30T13:00:09.049452+0000","last_clean":"2024-03-30T13:00:09.049452+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T13:00:09.049452+0000","last_undegraded":"2024-03-30T13:00:09.049452+0000","last_fullsized":"2024-03-30T13:00:09.049452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T12:59:28.587085+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T12:59:28.587085+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:52.894911+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":258,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-30T13:00:08.435905+0000","last_change":"2024-03-30T12:59:26.889088+0000","last_active":"2024-03-30T13:00:08.435905+0000","last_peered":"2024-03-30T13:00:08.435905+0000","last_clean":"2024-03-30T13:00:08.435905+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T13:00:08.435905+0000","last_undegraded":"2024-03-30T13:00:08.435905+0000","last_fullsized":"2024-03-30T13:00:08.435905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:59:26.888729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:26.888729+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:49.934982+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":37,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-30T13:00:07.636213+0000","last_change":"2024-03-30T12:59:55.354419+0000","last_active":"2024-03-30T13:00:07.636213+0000","last_peered":"2024-03-30T13:00:07.636213+0000","last_clean":"2024-03-30T13:00:07.636213+0000","last_became_active":"2024-03-30T12:59:55.310542+0000","last_became_peered":"2024-03-30T12:59:55.310542+0000","last_unstale":"2024-03-30T13:00:07.636213+0000","last_undegraded":"2024-03-30T13:00:07.636213+0000","last_fullsized":"2024-03-30T13:00:07.636213+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T12:59:24.867416+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:34.375550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":258,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-30T13:00:08.436028+0000","last_change":"2024-03-30T12:59:13.956465+0000","last_active":"2024-03-30T13:00:08.436028+0000","last_peered":"2024-03-30T13:00:08.436028+0000","last_clean":"2024-03-30T13:00:08.436028+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T13:00:08.436028+0000","last_undegraded":"2024-03-30T13:00:08.436028+0000","last_fullsized":"2024-03-30T13:00:08.436028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:59:13.956186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:13.956186+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:35.452148+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":238,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-30T13:00:07.424561+0000","last_change":"2024-03-30T13:00:06.700929+0000","last_active":"2024-03-30T13:00:07.424561+0000","last_peered":"2024-03-30T13:00:07.424561+0000","last_clean":"2024-03-30T13:00:07.424561+0000","last_became_active":"2024-03-30T12:58:59.113215+0000","last_became_peered":"2024-03-30T12:58:59.113215+0000","last_unstale":"2024-03-30T13:00:07.424561+0000","last_undegraded":"2024-03-30T13:00:07.424561+0000","last_fullsized":"2024-03-30T13:00:07.424561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:00:06.700643+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:58:34.875943+0000","last_clean_scrub_stamp":"2024-03-30T13:00:06.700643+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:25.056072+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":92286016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":95059968,"data_stored":95059968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509498,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43556,"kb_used_data":16536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94328284,"statfs":{"total":96636764160,"available":96592162816,"internally_reserved":0,"allocated":16932864,"data_stored":16111201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542202,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47084,"kb_used_data":20060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94324756,"statfs":{"total":96636764160,"available":96588550144,"internally_reserved":0,"allocated":20541440,"data_stored":19719777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822691,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44616,"kb_used_data":12924,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94327224,"statfs":{"total":96636764160,"available":96591077376,"internally_reserved":0,"allocated":13234176,"data_stored":12446518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4399,"internal_metadata":32435921},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509499,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34776,"kb_used_data":7752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337064,"statfs":{"total":96636764160,"available":96601153536,"internally_reserved":0,"allocated":7938048,"data_stored":7116385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285584,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34120,"kb_used_data":6584,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94337720,"statfs":{"total":96636764160,"available":96601825280,"internally_reserved":0,"allocated":6742016,"data_stored":5927154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":28176082},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509499,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41936,"kb_used_data":14916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329904,"statfs":{"total":96636764160,"available":96593821696,"internally_reserved":0,"allocated":15273984,"data_stored":14459122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":107,"seq":459561500678,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43612,"kb_used_data":11460,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94328228,"statfs":{"total":96636764160,"available":96592105472,"internally_reserved":0,"allocated":11735040,"data_stored":10920178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6042,"internal_metadata":32893030},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574907,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40932,"kb_used_data":13904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330908,"statfs":{"total":96636764160,"available":96594849792,"internally_reserved":0,"allocated":14237696,"data_stored":13429635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19079168,"data_stored":19079168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:00:10.824 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:00:10.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:10.938+0000 7fc081d46640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:10.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:10.950+0000 7fc081d46640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:11.128 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:00:11.128 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:00:11.146 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-03-30T13:00:10.461498+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":92286016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":330632,"kb_used_data":104136,"kb_used_omap":22,"kb_used_meta":226345,"kb_avail":754644088,"statfs":{"total":773094113280,"available":772755546112,"internally_reserved":0,"allocated":106635264,"data_stored":100129970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22815,"internal_metadata":231778017},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.039870"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":263,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-30T13:00:09.049303+0000","last_change":"2024-03-30T12:59:56.673098+0000","last_active":"2024-03-30T13:00:09.049303+0000","last_peered":"2024-03-30T13:00:09.049303+0000","last_clean":"2024-03-30T13:00:09.049303+0000","last_became_active":"2024-03-30T12:57:16.931067+0000","last_became_peered":"2024-03-30T12:57:16.931067+0000","last_unstale":"2024-03-30T13:00:09.049303+0000","last_undegraded":"2024-03-30T13:00:09.049303+0000","last_fullsized":"2024-03-30T13:00:09.049303+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:59:56.672842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:56.672842+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:15.718248+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":263,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-30T13:00:05.403774+0000","last_change":"2024-03-30T12:59:55.341764+0000","last_active":"2024-03-30T13:00:05.403774+0000","last_peered":"2024-03-30T13:00:05.403774+0000","last_clean":"2024-03-30T13:00:05.403774+0000","last_became_active":"2024-03-30T12:59:55.307854+0000","last_became_peered":"2024-03-30T12:59:55.307854+0000","last_unstale":"2024-03-30T13:00:05.403774+0000","last_undegraded":"2024-03-30T13:00:05.403774+0000","last_fullsized":"2024-03-30T13:00:05.403774+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:59:48.880811+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T12:59:48.880811+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:04.063697+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":13277365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":41,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-30T13:00:07.636309+0000","last_change":"2024-03-30T12:59:55.369991+0000","last_active":"2024-03-30T13:00:07.636309+0000","last_peered":"2024-03-30T13:00:07.636309+0000","last_clean":"2024-03-30T13:00:07.636309+0000","last_became_active":"2024-03-30T12:59:55.313597+0000","last_became_peered":"2024-03-30T12:59:55.313597+0000","last_unstale":"2024-03-30T13:00:07.636309+0000","last_undegraded":"2024-03-30T13:00:07.636309+0000","last_fullsized":"2024-03-30T13:00:07.636309+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:59:47.146312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:47.146312+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:59.164559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":39,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-30T13:00:07.636213+0000","last_change":"2024-03-30T12:59:55.395203+0000","last_active":"2024-03-30T13:00:07.636213+0000","last_peered":"2024-03-30T13:00:07.636213+0000","last_clean":"2024-03-30T13:00:07.636213+0000","last_became_active":"2024-03-30T12:59:55.316713+0000","last_became_peered":"2024-03-30T12:59:55.316713+0000","last_unstale":"2024-03-30T13:00:07.636213+0000","last_undegraded":"2024-03-30T13:00:07.636213+0000","last_fullsized":"2024-03-30T13:00:07.636213+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:59:20.897870+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T12:59:20.897870+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:30.841457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":246,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-30T13:00:08.435905+0000","last_change":"2024-03-30T13:00:01.952302+0000","last_active":"2024-03-30T13:00:08.435905+0000","last_peered":"2024-03-30T13:00:08.435905+0000","last_clean":"2024-03-30T13:00:08.435905+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T13:00:08.435905+0000","last_undegraded":"2024-03-30T13:00:08.435905+0000","last_fullsized":"2024-03-30T13:00:08.435905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:00:01.951942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:01.951942+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:08.852178+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":255,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-30T13:00:07.636104+0000","last_change":"2024-03-30T12:59:32.471464+0000","last_active":"2024-03-30T13:00:07.636104+0000","last_peered":"2024-03-30T13:00:07.636104+0000","last_clean":"2024-03-30T13:00:07.636104+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T13:00:07.636104+0000","last_undegraded":"2024-03-30T13:00:07.636104+0000","last_fullsized":"2024-03-30T13:00:07.636104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:59:32.471198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:32.471198+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:45.757823+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":54,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-30T13:00:07.636218+0000","last_change":"2024-03-30T12:59:55.487031+0000","last_active":"2024-03-30T13:00:07.636218+0000","last_peered":"2024-03-30T13:00:07.636218+0000","last_clean":"2024-03-30T13:00:07.636218+0000","last_became_active":"2024-03-30T12:59:55.317071+0000","last_became_peered":"2024-03-30T12:59:55.317071+0000","last_unstale":"2024-03-30T13:00:07.636218+0000","last_undegraded":"2024-03-30T13:00:07.636218+0000","last_fullsized":"2024-03-30T13:00:07.636218+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T12:59:20.919233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:20.919233+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:21.734360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":248,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-30T13:00:07.424478+0000","last_change":"2024-03-30T12:59:59.675214+0000","last_active":"2024-03-30T13:00:07.424478+0000","last_peered":"2024-03-30T13:00:07.424478+0000","last_clean":"2024-03-30T13:00:07.424478+0000","last_became_active":"2024-03-30T12:58:59.114041+0000","last_became_peered":"2024-03-30T12:58:59.114041+0000","last_unstale":"2024-03-30T13:00:07.424478+0000","last_undegraded":"2024-03-30T13:00:07.424478+0000","last_fullsized":"2024-03-30T13:00:07.424478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:59:59.675050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:59.675050+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:20.917814+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7344560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":297,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-30T13:00:07.636358+0000","last_change":"2024-03-30T12:59:55.347262+0000","last_active":"2024-03-30T13:00:07.636358+0000","last_peered":"2024-03-30T13:00:07.636358+0000","last_clean":"2024-03-30T13:00:07.636358+0000","last_became_active":"2024-03-30T12:59:55.305411+0000","last_became_peered":"2024-03-30T12:59:55.305411+0000","last_unstale":"2024-03-30T13:00:07.636358+0000","last_undegraded":"2024-03-30T13:00:07.636358+0000","last_fullsized":"2024-03-30T13:00:07.636358+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:59:18.470423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:18.470423+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:46.794865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":20302592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":212,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-03-30T13:00:05.403396+0000","last_change":"2024-03-30T12:59:42.816701+0000","last_active":"2024-03-30T13:00:05.403396+0000","last_peered":"2024-03-30T13:00:05.403396+0000","last_clean":"2024-03-30T13:00:05.403396+0000","last_became_active":"2024-03-30T12:57:16.930714+0000","last_became_peered":"2024-03-30T12:57:16.930714+0000","last_unstale":"2024-03-30T13:00:05.403396+0000","last_undegraded":"2024-03-30T13:00:05.403396+0000","last_fullsized":"2024-03-30T13:00:05.403396+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T12:59:42.816452+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:57.970457+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":285,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-30T13:00:09.049160+0000","last_change":"2024-03-30T12:59:37.603807+0000","last_active":"2024-03-30T13:00:09.049160+0000","last_peered":"2024-03-30T13:00:09.049160+0000","last_clean":"2024-03-30T13:00:09.049160+0000","last_became_active":"2024-03-30T12:57:16.930513+0000","last_became_peered":"2024-03-30T12:57:16.930513+0000","last_unstale":"2024-03-30T13:00:09.049160+0000","last_undegraded":"2024-03-30T13:00:09.049160+0000","last_fullsized":"2024-03-30T13:00:09.049160+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:59:37.603454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:37.603454+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:56.515014+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":270,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-30T13:00:09.049452+0000","last_change":"2024-03-30T12:59:28.587515+0000","last_active":"2024-03-30T13:00:09.049452+0000","last_peered":"2024-03-30T13:00:09.049452+0000","last_clean":"2024-03-30T13:00:09.049452+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T13:00:09.049452+0000","last_undegraded":"2024-03-30T13:00:09.049452+0000","last_fullsized":"2024-03-30T13:00:09.049452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T12:59:28.587085+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T12:59:28.587085+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:52.894911+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":258,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-30T13:00:08.435905+0000","last_change":"2024-03-30T12:59:26.889088+0000","last_active":"2024-03-30T13:00:08.435905+0000","last_peered":"2024-03-30T13:00:08.435905+0000","last_clean":"2024-03-30T13:00:08.435905+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T13:00:08.435905+0000","last_undegraded":"2024-03-30T13:00:08.435905+0000","last_fullsized":"2024-03-30T13:00:08.435905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:59:26.888729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:26.888729+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:49.934982+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":37,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-30T13:00:07.636213+0000","last_change":"2024-03-30T12:59:55.354419+0000","last_active":"2024-03-30T13:00:07.636213+0000","last_peered":"2024-03-30T13:00:07.636213+0000","last_clean":"2024-03-30T13:00:07.636213+0000","last_became_active":"2024-03-30T12:59:55.310542+0000","last_became_peered":"2024-03-30T12:59:55.310542+0000","last_unstale":"2024-03-30T13:00:07.636213+0000","last_undegraded":"2024-03-30T13:00:07.636213+0000","last_fullsized":"2024-03-30T13:00:07.636213+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T12:59:24.867416+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:34.375550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":258,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-30T13:00:08.436028+0000","last_change":"2024-03-30T12:59:13.956465+0000","last_active":"2024-03-30T13:00:08.436028+0000","last_peered":"2024-03-30T13:00:08.436028+0000","last_clean":"2024-03-30T13:00:08.436028+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T13:00:08.436028+0000","last_undegraded":"2024-03-30T13:00:08.436028+0000","last_fullsized":"2024-03-30T13:00:08.436028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:59:13.956186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:13.956186+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:35.452148+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":238,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-30T13:00:07.424561+0000","last_change":"2024-03-30T13:00:06.700929+0000","last_active":"2024-03-30T13:00:07.424561+0000","last_peered":"2024-03-30T13:00:07.424561+0000","last_clean":"2024-03-30T13:00:07.424561+0000","last_became_active":"2024-03-30T12:58:59.113215+0000","last_became_peered":"2024-03-30T12:58:59.113215+0000","last_unstale":"2024-03-30T13:00:07.424561+0000","last_undegraded":"2024-03-30T13:00:07.424561+0000","last_fullsized":"2024-03-30T13:00:07.424561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:00:06.700643+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:58:34.875943+0000","last_clean_scrub_stamp":"2024-03-30T13:00:06.700643+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:25.056072+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":92286016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":95059968,"data_stored":95059968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509498,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43556,"kb_used_data":16536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94328284,"statfs":{"total":96636764160,"available":96592162816,"internally_reserved":0,"allocated":16932864,"data_stored":16111201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542202,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47084,"kb_used_data":20060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94324756,"statfs":{"total":96636764160,"available":96588550144,"internally_reserved":0,"allocated":20541440,"data_stored":19719777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822691,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44616,"kb_used_data":12924,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94327224,"statfs":{"total":96636764160,"available":96591077376,"internally_reserved":0,"allocated":13234176,"data_stored":12446518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4399,"internal_metadata":32435921},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509499,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34776,"kb_used_data":7752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337064,"statfs":{"total":96636764160,"available":96601153536,"internally_reserved":0,"allocated":7938048,"data_stored":7116385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285584,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34120,"kb_used_data":6584,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94337720,"statfs":{"total":96636764160,"available":96601825280,"internally_reserved":0,"allocated":6742016,"data_stored":5927154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":28176082},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509499,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41936,"kb_used_data":14916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329904,"statfs":{"total":96636764160,"available":96593821696,"internally_reserved":0,"allocated":15273984,"data_stored":14459122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":107,"seq":459561500678,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43612,"kb_used_data":11460,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94328228,"statfs":{"total":96636764160,"available":96592105472,"internally_reserved":0,"allocated":11735040,"data_stored":10920178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6042,"internal_metadata":32893030},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574907,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40932,"kb_used_data":13904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330908,"statfs":{"total":96636764160,"available":96594849792,"internally_reserved":0,"allocated":14237696,"data_stored":13429635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19079168,"data_stored":19079168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:00:11.148 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:00:11.148 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:00:11.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:11.266+0000 7f2b59d12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:11.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:11.278+0000 7f2b59d12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:11.455 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:00:11.455 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":125,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:00:10.457649+0000","last_up_change":"2024-03-30T12:59:49.213511+0000","last_in_change":"2024-03-30T12:59:54.275507+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":10,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":77,"up_from":83,"up_thru":102,"down_at":78,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6818","nonce":3104224893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6821","nonce":3104224893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6828","nonce":3104224893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6825","nonce":3104224893}]},"public_addr":"172.21.15.67:6818/3104224893","cluster_addr":"172.21.15.67:6821/3104224893","heartbeat_back_addr":"172.21.15.67:6828/3104224893","heartbeat_front_addr":"172.21.15.67:6825/3104224893","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6801","nonce":1504348724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6803","nonce":1504348724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6807","nonce":1504348724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6805","nonce":1504348724}]},"public_addr":"172.21.15.67:6801/1504348724","cluster_addr":"172.21.15.67:6803/1504348724","heartbeat_back_addr":"172.21.15.67:6807/1504348724","heartbeat_front_addr":"172.21.15.67:6805/1504348724","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":25,"up_from":36,"up_thru":109,"down_at":26,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1927174653},{"type":"v1","addr":"172.21.15.82:6803","nonce":1927174653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":1927174653},{"type":"v1","addr":"172.21.15.82:6807","nonce":1927174653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":1927174653},{"type":"v1","addr":"172.21.15.82:6815","nonce":1927174653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":1927174653},{"type":"v1","addr":"172.21.15.82:6811","nonce":1927174653}]},"public_addr":"172.21.15.82:6803/1927174653","cluster_addr":"172.21.15.82:6807/1927174653","heartbeat_back_addr":"172.21.15.82:6815/1927174653","heartbeat_front_addr":"172.21.15.82:6811/1927174653","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6817","nonce":3467856022}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6819","nonce":3467856022}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6823","nonce":3467856022}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6821","nonce":3467856022}]},"public_addr":"172.21.15.82:6817/3467856022","cluster_addr":"172.21.15.82:6819/3467856022","heartbeat_back_addr":"172.21.15.82:6823/3467856022","heartbeat_front_addr":"172.21.15.82:6821/3467856022","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6801","nonce":968508548}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6806","nonce":968508548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6814","nonce":968508548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6810","nonce":968508548}]},"public_addr":"172.21.15.82:6801/968508548","cluster_addr":"172.21.15.82:6806/968508548","heartbeat_back_addr":"172.21.15.82:6814/968508548","heartbeat_front_addr":"172.21.15.82:6810/968508548","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":100,"up_from":107,"up_thru":109,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2568135668},{"type":"v1","addr":"172.21.15.82:6825","nonce":2568135668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2568135668},{"type":"v1","addr":"172.21.15.82:6827","nonce":2568135668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2568135668},{"type":"v1","addr":"172.21.15.82:6831","nonce":2568135668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2568135668},{"type":"v1","addr":"172.21.15.82:6829","nonce":2568135668}]},"public_addr":"172.21.15.82:6825/2568135668","cluster_addr":"172.21.15.82:6827/2568135668","heartbeat_back_addr":"172.21.15.82:6831/2568135668","heartbeat_front_addr":"172.21.15.82:6829/2568135668","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:59:14.803877+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:59:08.569659+0000","dead_epoch":78},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:59:14.544097+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:59:19.273471+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:59:45.437353+0000","dead_epoch":26},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:59:16.925606+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:59:15.291522+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T12:59:48.244063+0000","dead_epoch":101}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:00:11.474 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:00:11.474 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:00:11.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:11.590+0000 7f1025d6b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:11.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:11.606+0000 7f1025d6b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:11.778 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:00:11.798 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:00:11.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:11.918+0000 7ff5b2548640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:11.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:11.934+0000 7ff5b2548640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:12.117 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:00:12.118 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:00:12.137 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:00:12.137 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:00:12.137 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:00:12.137 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:00:12.137 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 3, 5, 6, 4, 1, 7] 2024-03-30T13:00:12.138 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:00:12.138 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:00:12.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:12.250+0000 7f9be68a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:12.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:12.262+0000 7f9be68a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:12.437 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:00:12.438 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:00:12.455 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":258,"stamp":"2024-03-30T13:00:11.861797+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":92286016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":330704,"kb_used_data":104208,"kb_used_omap":22,"kb_used_meta":226345,"kb_avail":754644016,"statfs":{"total":773094113280,"available":772755472384,"internally_reserved":0,"allocated":106708992,"data_stored":100162892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22815,"internal_metadata":231778017},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.437574"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":263,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-30T13:00:09.049303+0000","last_change":"2024-03-30T12:59:56.673098+0000","last_active":"2024-03-30T13:00:09.049303+0000","last_peered":"2024-03-30T13:00:09.049303+0000","last_clean":"2024-03-30T13:00:09.049303+0000","last_became_active":"2024-03-30T12:57:16.931067+0000","last_became_peered":"2024-03-30T12:57:16.931067+0000","last_unstale":"2024-03-30T13:00:09.049303+0000","last_undegraded":"2024-03-30T13:00:09.049303+0000","last_fullsized":"2024-03-30T13:00:09.049303+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:59:56.672842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:56.672842+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:15.718248+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":273,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-30T13:00:10.597197+0000","last_change":"2024-03-30T12:59:55.341764+0000","last_active":"2024-03-30T13:00:10.597197+0000","last_peered":"2024-03-30T13:00:10.597197+0000","last_clean":"2024-03-30T13:00:10.597197+0000","last_became_active":"2024-03-30T12:59:55.307854+0000","last_became_peered":"2024-03-30T12:59:55.307854+0000","last_unstale":"2024-03-30T13:00:10.597197+0000","last_undegraded":"2024-03-30T13:00:10.597197+0000","last_fullsized":"2024-03-30T13:00:10.597197+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:59:48.880811+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T12:59:48.880811+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:04.063697+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":13277365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":41,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-30T13:00:07.636309+0000","last_change":"2024-03-30T12:59:55.369991+0000","last_active":"2024-03-30T13:00:07.636309+0000","last_peered":"2024-03-30T13:00:07.636309+0000","last_clean":"2024-03-30T13:00:07.636309+0000","last_became_active":"2024-03-30T12:59:55.313597+0000","last_became_peered":"2024-03-30T12:59:55.313597+0000","last_unstale":"2024-03-30T13:00:07.636309+0000","last_undegraded":"2024-03-30T13:00:07.636309+0000","last_fullsized":"2024-03-30T13:00:07.636309+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:59:47.146312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:47.146312+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:59.164559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":39,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-30T13:00:07.636213+0000","last_change":"2024-03-30T12:59:55.395203+0000","last_active":"2024-03-30T13:00:07.636213+0000","last_peered":"2024-03-30T13:00:07.636213+0000","last_clean":"2024-03-30T13:00:07.636213+0000","last_became_active":"2024-03-30T12:59:55.316713+0000","last_became_peered":"2024-03-30T12:59:55.316713+0000","last_unstale":"2024-03-30T13:00:07.636213+0000","last_undegraded":"2024-03-30T13:00:07.636213+0000","last_fullsized":"2024-03-30T13:00:07.636213+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:59:20.897870+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T12:59:20.897870+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:30.841457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":246,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-30T13:00:08.435905+0000","last_change":"2024-03-30T13:00:01.952302+0000","last_active":"2024-03-30T13:00:08.435905+0000","last_peered":"2024-03-30T13:00:08.435905+0000","last_clean":"2024-03-30T13:00:08.435905+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T13:00:08.435905+0000","last_undegraded":"2024-03-30T13:00:08.435905+0000","last_fullsized":"2024-03-30T13:00:08.435905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:00:01.951942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:01.951942+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:08.852178+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":255,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-30T13:00:07.636104+0000","last_change":"2024-03-30T12:59:32.471464+0000","last_active":"2024-03-30T13:00:07.636104+0000","last_peered":"2024-03-30T13:00:07.636104+0000","last_clean":"2024-03-30T13:00:07.636104+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T13:00:07.636104+0000","last_undegraded":"2024-03-30T13:00:07.636104+0000","last_fullsized":"2024-03-30T13:00:07.636104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:59:32.471198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:32.471198+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:45.757823+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":54,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-30T13:00:07.636218+0000","last_change":"2024-03-30T12:59:55.487031+0000","last_active":"2024-03-30T13:00:07.636218+0000","last_peered":"2024-03-30T13:00:07.636218+0000","last_clean":"2024-03-30T13:00:07.636218+0000","last_became_active":"2024-03-30T12:59:55.317071+0000","last_became_peered":"2024-03-30T12:59:55.317071+0000","last_unstale":"2024-03-30T13:00:07.636218+0000","last_undegraded":"2024-03-30T13:00:07.636218+0000","last_fullsized":"2024-03-30T13:00:07.636218+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T12:59:20.919233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:20.919233+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:21.734360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":248,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-30T13:00:07.424478+0000","last_change":"2024-03-30T12:59:59.675214+0000","last_active":"2024-03-30T13:00:07.424478+0000","last_peered":"2024-03-30T13:00:07.424478+0000","last_clean":"2024-03-30T13:00:07.424478+0000","last_became_active":"2024-03-30T12:58:59.114041+0000","last_became_peered":"2024-03-30T12:58:59.114041+0000","last_unstale":"2024-03-30T13:00:07.424478+0000","last_undegraded":"2024-03-30T13:00:07.424478+0000","last_fullsized":"2024-03-30T13:00:07.424478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:59:59.675050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:59.675050+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:20.917814+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7344560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":297,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-30T13:00:07.636358+0000","last_change":"2024-03-30T12:59:55.347262+0000","last_active":"2024-03-30T13:00:07.636358+0000","last_peered":"2024-03-30T13:00:07.636358+0000","last_clean":"2024-03-30T13:00:07.636358+0000","last_became_active":"2024-03-30T12:59:55.305411+0000","last_became_peered":"2024-03-30T12:59:55.305411+0000","last_unstale":"2024-03-30T13:00:07.636358+0000","last_undegraded":"2024-03-30T13:00:07.636358+0000","last_fullsized":"2024-03-30T13:00:07.636358+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:59:18.470423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:18.470423+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:46.794865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":20302592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":222,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-30T13:00:10.597093+0000","last_change":"2024-03-30T12:59:42.816701+0000","last_active":"2024-03-30T13:00:10.597093+0000","last_peered":"2024-03-30T13:00:10.597093+0000","last_clean":"2024-03-30T13:00:10.597093+0000","last_became_active":"2024-03-30T12:57:16.930714+0000","last_became_peered":"2024-03-30T12:57:16.930714+0000","last_unstale":"2024-03-30T13:00:10.597093+0000","last_undegraded":"2024-03-30T13:00:10.597093+0000","last_fullsized":"2024-03-30T13:00:10.597093+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T12:59:42.816452+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:57.970457+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":285,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-30T13:00:09.049160+0000","last_change":"2024-03-30T12:59:37.603807+0000","last_active":"2024-03-30T13:00:09.049160+0000","last_peered":"2024-03-30T13:00:09.049160+0000","last_clean":"2024-03-30T13:00:09.049160+0000","last_became_active":"2024-03-30T12:57:16.930513+0000","last_became_peered":"2024-03-30T12:57:16.930513+0000","last_unstale":"2024-03-30T13:00:09.049160+0000","last_undegraded":"2024-03-30T13:00:09.049160+0000","last_fullsized":"2024-03-30T13:00:09.049160+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:59:37.603454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:37.603454+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:56.515014+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":270,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-30T13:00:09.049452+0000","last_change":"2024-03-30T12:59:28.587515+0000","last_active":"2024-03-30T13:00:09.049452+0000","last_peered":"2024-03-30T13:00:09.049452+0000","last_clean":"2024-03-30T13:00:09.049452+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T13:00:09.049452+0000","last_undegraded":"2024-03-30T13:00:09.049452+0000","last_fullsized":"2024-03-30T13:00:09.049452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T12:59:28.587085+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T12:59:28.587085+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:52.894911+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":258,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-30T13:00:08.435905+0000","last_change":"2024-03-30T12:59:26.889088+0000","last_active":"2024-03-30T13:00:08.435905+0000","last_peered":"2024-03-30T13:00:08.435905+0000","last_clean":"2024-03-30T13:00:08.435905+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T13:00:08.435905+0000","last_undegraded":"2024-03-30T13:00:08.435905+0000","last_fullsized":"2024-03-30T13:00:08.435905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:59:26.888729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:26.888729+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:49.934982+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":37,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-30T13:00:07.636213+0000","last_change":"2024-03-30T12:59:55.354419+0000","last_active":"2024-03-30T13:00:07.636213+0000","last_peered":"2024-03-30T13:00:07.636213+0000","last_clean":"2024-03-30T13:00:07.636213+0000","last_became_active":"2024-03-30T12:59:55.310542+0000","last_became_peered":"2024-03-30T12:59:55.310542+0000","last_unstale":"2024-03-30T13:00:07.636213+0000","last_undegraded":"2024-03-30T13:00:07.636213+0000","last_fullsized":"2024-03-30T13:00:07.636213+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T12:59:24.867416+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:34.375550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":258,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-30T13:00:08.436028+0000","last_change":"2024-03-30T12:59:13.956465+0000","last_active":"2024-03-30T13:00:08.436028+0000","last_peered":"2024-03-30T13:00:08.436028+0000","last_clean":"2024-03-30T13:00:08.436028+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T13:00:08.436028+0000","last_undegraded":"2024-03-30T13:00:08.436028+0000","last_fullsized":"2024-03-30T13:00:08.436028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:59:13.956186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:13.956186+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:35.452148+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":238,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-03-30T13:00:07.424561+0000","last_change":"2024-03-30T13:00:06.700929+0000","last_active":"2024-03-30T13:00:07.424561+0000","last_peered":"2024-03-30T13:00:07.424561+0000","last_clean":"2024-03-30T13:00:07.424561+0000","last_became_active":"2024-03-30T12:58:59.113215+0000","last_became_peered":"2024-03-30T12:58:59.113215+0000","last_unstale":"2024-03-30T13:00:07.424561+0000","last_undegraded":"2024-03-30T13:00:07.424561+0000","last_fullsized":"2024-03-30T13:00:07.424561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:00:06.700643+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:58:34.875943+0000","last_clean_scrub_stamp":"2024-03-30T13:00:06.700643+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:25.056072+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":92286016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":95059968,"data_stored":95059968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509498,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43556,"kb_used_data":16536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94328284,"statfs":{"total":96636764160,"available":96592162816,"internally_reserved":0,"allocated":16932864,"data_stored":16111201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542202,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47084,"kb_used_data":20060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94324756,"statfs":{"total":96636764160,"available":96588550144,"internally_reserved":0,"allocated":20541440,"data_stored":19719777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":36,"seq":154618822692,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44688,"kb_used_data":12996,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94327152,"statfs":{"total":96636764160,"available":96591003648,"internally_reserved":0,"allocated":13307904,"data_stored":12479440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4399,"internal_metadata":32435921},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509499,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34776,"kb_used_data":7752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337064,"statfs":{"total":96636764160,"available":96601153536,"internally_reserved":0,"allocated":7938048,"data_stored":7116385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285584,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34120,"kb_used_data":6584,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94337720,"statfs":{"total":96636764160,"available":96601825280,"internally_reserved":0,"allocated":6742016,"data_stored":5927154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":28176082},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509499,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41936,"kb_used_data":14916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329904,"statfs":{"total":96636764160,"available":96593821696,"internally_reserved":0,"allocated":15273984,"data_stored":14459122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":107,"seq":459561500678,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43612,"kb_used_data":11460,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94328228,"statfs":{"total":96636764160,"available":96592105472,"internally_reserved":0,"allocated":11735040,"data_stored":10920178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6042,"internal_metadata":32893030},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574907,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40932,"kb_used_data":13904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330908,"statfs":{"total":96636764160,"available":96594849792,"internally_reserved":0,"allocated":14237696,"data_stored":13429635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19079168,"data_stored":19079168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:00:12.456 INFO:tasks.thrashosds.thrasher:Choosing PG 1.5 with acting set [4, 2, 6] 2024-03-30T13:00:12.456 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [0, 2, 3, 5, 6, 4, 1, 7] 2024-03-30T13:00:12.457 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-30T13:00:12.457 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:00:12.524 INFO:tasks.ceph.osd.4:Stopped 2024-03-30T13:00:12.524 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 2, 3, 5, 6, 4, 1, 7] 2024-03-30T13:00:12.524 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 4 2024-03-30T13:00:12.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:12.638+0000 7f2b16a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:12.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:12.654+0000 7f2b16a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:12.836 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:00:12.834+0000 7fbb38f08640 -1 mon.a@0(leader).osd e125 definitely_dead 0 2024-03-30T13:00:13.501 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.4. 2024-03-30T13:00:13.532 INFO:tasks.thrashosds.thrasher:dead_osds=[4], live_osds=[0, 2, 3, 5, 6, 1, 7] 2024-03-30T13:00:13.532 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:00:13.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:00:13.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:13.650+0000 7f1f292a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:13.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:13.666+0000 7f1f292a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:13.840 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:00:13.841 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:00:13.859 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":259,"stamp":"2024-03-30T13:00:13.502950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":92286016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":286128,"kb_used_data":91332,"kb_used_omap":17,"kb_used_meta":194670,"kb_avail":660316752,"statfs":{"total":676457349120,"available":676164354048,"internally_reserved":0,"allocated":93523968,"data_stored":87738322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18416,"internal_metadata":199342096},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.642275"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":263,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-30T13:00:09.049303+0000","last_change":"2024-03-30T12:59:56.673098+0000","last_active":"2024-03-30T13:00:09.049303+0000","last_peered":"2024-03-30T13:00:09.049303+0000","last_clean":"2024-03-30T13:00:09.049303+0000","last_became_active":"2024-03-30T12:57:16.931067+0000","last_became_peered":"2024-03-30T12:57:16.931067+0000","last_unstale":"2024-03-30T13:00:09.049303+0000","last_undegraded":"2024-03-30T13:00:09.049303+0000","last_fullsized":"2024-03-30T13:00:09.049303+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:59:56.672842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:56.672842+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:15.718248+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":273,"reported_epoch":125,"state":"stale+active+clean","last_fresh":"2024-03-30T13:00:10.597197+0000","last_change":"2024-03-30T12:59:55.341764+0000","last_active":"2024-03-30T13:00:10.597197+0000","last_peered":"2024-03-30T13:00:10.597197+0000","last_clean":"2024-03-30T13:00:10.597197+0000","last_became_active":"2024-03-30T12:59:55.307854+0000","last_became_peered":"2024-03-30T12:59:55.307854+0000","last_unstale":"2024-03-30T13:00:13.502972+0000","last_undegraded":"2024-03-30T13:00:10.597197+0000","last_fullsized":"2024-03-30T13:00:10.597197+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:59:48.880811+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T12:59:48.880811+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:04.063697+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":13277365,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":47,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-30T13:00:10.586764+0000","last_change":"2024-03-30T12:59:55.369991+0000","last_active":"2024-03-30T13:00:10.586764+0000","last_peered":"2024-03-30T13:00:10.586764+0000","last_clean":"2024-03-30T13:00:10.586764+0000","last_became_active":"2024-03-30T12:59:55.313597+0000","last_became_peered":"2024-03-30T12:59:55.313597+0000","last_unstale":"2024-03-30T13:00:10.586764+0000","last_undegraded":"2024-03-30T13:00:10.586764+0000","last_fullsized":"2024-03-30T13:00:10.586764+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:59:47.146312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:47.146312+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:59.164559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":45,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-30T13:00:10.586659+0000","last_change":"2024-03-30T12:59:55.395203+0000","last_active":"2024-03-30T13:00:10.586659+0000","last_peered":"2024-03-30T13:00:10.586659+0000","last_clean":"2024-03-30T13:00:10.586659+0000","last_became_active":"2024-03-30T12:59:55.316713+0000","last_became_peered":"2024-03-30T12:59:55.316713+0000","last_unstale":"2024-03-30T13:00:10.586659+0000","last_undegraded":"2024-03-30T13:00:10.586659+0000","last_fullsized":"2024-03-30T13:00:10.586659+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:59:20.897870+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T12:59:20.897870+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:30.841457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":246,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-30T13:00:08.435905+0000","last_change":"2024-03-30T13:00:01.952302+0000","last_active":"2024-03-30T13:00:08.435905+0000","last_peered":"2024-03-30T13:00:08.435905+0000","last_clean":"2024-03-30T13:00:08.435905+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T13:00:08.435905+0000","last_undegraded":"2024-03-30T13:00:08.435905+0000","last_fullsized":"2024-03-30T13:00:08.435905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:00:01.951942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:01.951942+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:08.852178+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":261,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-30T13:00:10.748989+0000","last_change":"2024-03-30T12:59:32.471464+0000","last_active":"2024-03-30T13:00:10.748989+0000","last_peered":"2024-03-30T13:00:10.748989+0000","last_clean":"2024-03-30T13:00:10.748989+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T13:00:10.748989+0000","last_undegraded":"2024-03-30T13:00:10.748989+0000","last_fullsized":"2024-03-30T13:00:10.748989+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:59:32.471198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:32.471198+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:45.757823+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":60,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-30T13:00:10.586687+0000","last_change":"2024-03-30T12:59:55.487031+0000","last_active":"2024-03-30T13:00:10.586687+0000","last_peered":"2024-03-30T13:00:10.586687+0000","last_clean":"2024-03-30T13:00:10.586687+0000","last_became_active":"2024-03-30T12:59:55.317071+0000","last_became_peered":"2024-03-30T12:59:55.317071+0000","last_unstale":"2024-03-30T13:00:10.586687+0000","last_undegraded":"2024-03-30T13:00:10.586687+0000","last_fullsized":"2024-03-30T13:00:10.586687+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T12:59:20.919233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:20.919233+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:21.734360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":254,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-30T13:00:10.463263+0000","last_change":"2024-03-30T12:59:59.675214+0000","last_active":"2024-03-30T13:00:10.463263+0000","last_peered":"2024-03-30T13:00:10.463263+0000","last_clean":"2024-03-30T13:00:10.463263+0000","last_became_active":"2024-03-30T12:58:59.114041+0000","last_became_peered":"2024-03-30T12:58:59.114041+0000","last_unstale":"2024-03-30T13:00:10.463263+0000","last_undegraded":"2024-03-30T13:00:10.463263+0000","last_fullsized":"2024-03-30T13:00:10.463263+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:59:59.675050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:59.675050+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:20.917814+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7344560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":303,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-30T13:00:10.749291+0000","last_change":"2024-03-30T12:59:55.347262+0000","last_active":"2024-03-30T13:00:10.749291+0000","last_peered":"2024-03-30T13:00:10.749291+0000","last_clean":"2024-03-30T13:00:10.749291+0000","last_became_active":"2024-03-30T12:59:55.305411+0000","last_became_peered":"2024-03-30T12:59:55.305411+0000","last_unstale":"2024-03-30T13:00:10.749291+0000","last_undegraded":"2024-03-30T13:00:10.749291+0000","last_fullsized":"2024-03-30T13:00:10.749291+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:59:18.470423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:18.470423+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:46.794865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":20302592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":222,"reported_epoch":125,"state":"stale+active+clean","last_fresh":"2024-03-30T13:00:10.597093+0000","last_change":"2024-03-30T12:59:42.816701+0000","last_active":"2024-03-30T13:00:10.597093+0000","last_peered":"2024-03-30T13:00:10.597093+0000","last_clean":"2024-03-30T13:00:10.597093+0000","last_became_active":"2024-03-30T12:57:16.930714+0000","last_became_peered":"2024-03-30T12:57:16.930714+0000","last_unstale":"2024-03-30T13:00:13.502976+0000","last_undegraded":"2024-03-30T13:00:10.597093+0000","last_fullsized":"2024-03-30T13:00:10.597093+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T12:59:42.816452+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:57.970457+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":285,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-30T13:00:09.049160+0000","last_change":"2024-03-30T12:59:37.603807+0000","last_active":"2024-03-30T13:00:09.049160+0000","last_peered":"2024-03-30T13:00:09.049160+0000","last_clean":"2024-03-30T13:00:09.049160+0000","last_became_active":"2024-03-30T12:57:16.930513+0000","last_became_peered":"2024-03-30T12:57:16.930513+0000","last_unstale":"2024-03-30T13:00:09.049160+0000","last_undegraded":"2024-03-30T13:00:09.049160+0000","last_fullsized":"2024-03-30T13:00:09.049160+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:59:37.603454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:37.603454+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:56.515014+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":270,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-30T13:00:09.049452+0000","last_change":"2024-03-30T12:59:28.587515+0000","last_active":"2024-03-30T13:00:09.049452+0000","last_peered":"2024-03-30T13:00:09.049452+0000","last_clean":"2024-03-30T13:00:09.049452+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T13:00:09.049452+0000","last_undegraded":"2024-03-30T13:00:09.049452+0000","last_fullsized":"2024-03-30T13:00:09.049452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T12:59:28.587085+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T12:59:28.587085+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:52.894911+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":258,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-30T13:00:08.435905+0000","last_change":"2024-03-30T12:59:26.889088+0000","last_active":"2024-03-30T13:00:08.435905+0000","last_peered":"2024-03-30T13:00:08.435905+0000","last_clean":"2024-03-30T13:00:08.435905+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T13:00:08.435905+0000","last_undegraded":"2024-03-30T13:00:08.435905+0000","last_fullsized":"2024-03-30T13:00:08.435905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:59:26.888729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:26.888729+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:49.934982+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":43,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-30T13:00:10.586719+0000","last_change":"2024-03-30T12:59:55.354419+0000","last_active":"2024-03-30T13:00:10.586719+0000","last_peered":"2024-03-30T13:00:10.586719+0000","last_clean":"2024-03-30T13:00:10.586719+0000","last_became_active":"2024-03-30T12:59:55.310542+0000","last_became_peered":"2024-03-30T12:59:55.310542+0000","last_unstale":"2024-03-30T13:00:10.586719+0000","last_undegraded":"2024-03-30T13:00:10.586719+0000","last_fullsized":"2024-03-30T13:00:10.586719+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T12:59:24.867416+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:34.375550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":258,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-03-30T13:00:08.436028+0000","last_change":"2024-03-30T12:59:13.956465+0000","last_active":"2024-03-30T13:00:08.436028+0000","last_peered":"2024-03-30T13:00:08.436028+0000","last_clean":"2024-03-30T13:00:08.436028+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T13:00:08.436028+0000","last_undegraded":"2024-03-30T13:00:08.436028+0000","last_fullsized":"2024-03-30T13:00:08.436028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:59:13.956186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:13.956186+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:35.452148+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":244,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-30T13:00:10.463327+0000","last_change":"2024-03-30T13:00:06.700929+0000","last_active":"2024-03-30T13:00:10.463327+0000","last_peered":"2024-03-30T13:00:10.463327+0000","last_clean":"2024-03-30T13:00:10.463327+0000","last_became_active":"2024-03-30T12:58:59.113215+0000","last_became_peered":"2024-03-30T12:58:59.113215+0000","last_unstale":"2024-03-30T13:00:10.463327+0000","last_undegraded":"2024-03-30T13:00:10.463327+0000","last_fullsized":"2024-03-30T13:00:10.463327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:00:06.700643+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:58:34.875943+0000","last_clean_scrub_stamp":"2024-03-30T13:00:06.700643+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:25.056072+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":92286016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":95059968,"data_stored":95059968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509498,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43556,"kb_used_data":16536,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94328284,"statfs":{"total":96636764160,"available":96592162816,"internally_reserved":0,"allocated":16932864,"data_stored":16111201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542202,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47084,"kb_used_data":20060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94324756,"statfs":{"total":96636764160,"available":96588550144,"internally_reserved":0,"allocated":20541440,"data_stored":19719777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509499,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34776,"kb_used_data":7752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337064,"statfs":{"total":96636764160,"available":96601153536,"internally_reserved":0,"allocated":7938048,"data_stored":7116385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285585,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34156,"kb_used_data":6620,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94337684,"statfs":{"total":96636764160,"available":96601788416,"internally_reserved":0,"allocated":6778880,"data_stored":5943615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":28176082},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509499,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41936,"kb_used_data":14916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329904,"statfs":{"total":96636764160,"available":96593821696,"internally_reserved":0,"allocated":15273984,"data_stored":14459122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":107,"seq":459561500679,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43644,"kb_used_data":11496,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94328196,"statfs":{"total":96636764160,"available":96592072704,"internally_reserved":0,"allocated":11771904,"data_stored":10936639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6042,"internal_metadata":32893030},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574908,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40976,"kb_used_data":13952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330864,"statfs":{"total":96636764160,"available":96594804736,"internally_reserved":0,"allocated":14286848,"data_stored":13451583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19079168,"data_stored":19079168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:00:13.860 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:00:38.862 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:00:38.862 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:00:39.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:38.989+0000 7facc920a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:39.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:39.005+0000 7facc920a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:39.192 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:00:39.194 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:00:39.209 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":278,"stamp":"2024-03-30T13:00:39.051180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":121445765,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":303080,"kb_used_data":104144,"kb_used_omap":17,"kb_used_meta":198830,"kb_avail":660299800,"statfs":{"total":676457349120,"available":676146995200,"internally_reserved":0,"allocated":106643456,"data_stored":100520350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18416,"internal_metadata":203601936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.184498"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":291,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-03-30T13:00:33.012845+0000","last_change":"2024-03-30T13:00:14.585704+0000","last_active":"2024-03-30T13:00:33.012845+0000","last_peered":"2024-03-30T13:00:33.012845+0000","last_clean":"2024-03-30T13:00:33.012845+0000","last_became_active":"2024-03-30T13:00:14.527030+0000","last_became_peered":"2024-03-30T13:00:14.527030+0000","last_unstale":"2024-03-30T13:00:33.012845+0000","last_undegraded":"2024-03-30T13:00:33.012845+0000","last_fullsized":"2024-03-30T13:00:33.012845+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:59:56.672842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:56.672842+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:58.106466+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":23,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-03-30T13:00:33.013953+0000","last_change":"2024-03-30T13:00:14.569165+0000","last_active":"2024-03-30T13:00:33.013953+0000","last_peered":"2024-03-30T13:00:33.013953+0000","last_clean":"2024-03-30T13:00:33.013953+0000","last_became_active":"2024-03-30T13:00:14.535571+0000","last_became_peered":"2024-03-30T13:00:14.535571+0000","last_unstale":"2024-03-30T13:00:33.013953+0000","last_undegraded":"2024-03-30T13:00:33.013953+0000","last_fullsized":"2024-03-30T13:00:33.013953+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:59:48.880811+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T12:59:48.880811+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:49.006002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":73,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-03-30T13:00:34.134998+0000","last_change":"2024-03-30T13:00:14.651400+0000","last_active":"2024-03-30T13:00:34.134998+0000","last_peered":"2024-03-30T13:00:34.134998+0000","last_clean":"2024-03-30T13:00:34.134998+0000","last_became_active":"2024-03-30T13:00:14.536038+0000","last_became_peered":"2024-03-30T13:00:14.536038+0000","last_unstale":"2024-03-30T13:00:34.134998+0000","last_undegraded":"2024-03-30T13:00:34.134998+0000","last_fullsized":"2024-03-30T13:00:34.134998+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:59:47.146312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:47.146312+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:08.547782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":65,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-03-30T13:00:35.159821+0000","last_change":"2024-03-30T13:00:35.159895+0000","last_active":"2024-03-30T13:00:35.159821+0000","last_peered":"2024-03-30T13:00:35.159821+0000","last_clean":"2024-03-30T13:00:35.159821+0000","last_became_active":"2024-03-30T12:59:55.316713+0000","last_became_peered":"2024-03-30T12:59:55.316713+0000","last_unstale":"2024-03-30T13:00:35.159821+0000","last_undegraded":"2024-03-30T13:00:35.159821+0000","last_fullsized":"2024-03-30T13:00:35.159821+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:00:35.159640+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:00:35.159640+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:04.854390+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":259,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-30T13:00:24.992090+0000","last_change":"2024-03-30T13:00:01.952302+0000","last_active":"2024-03-30T13:00:24.992090+0000","last_peered":"2024-03-30T13:00:24.992090+0000","last_clean":"2024-03-30T13:00:24.992090+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T13:00:24.992090+0000","last_undegraded":"2024-03-30T13:00:24.992090+0000","last_fullsized":"2024-03-30T13:00:24.992090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:00:01.951942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:01.951942+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:08.852178+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":274,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-03-30T13:00:34.322860+0000","last_change":"2024-03-30T12:59:32.471464+0000","last_active":"2024-03-30T13:00:34.322860+0000","last_peered":"2024-03-30T13:00:34.322860+0000","last_clean":"2024-03-30T13:00:34.322860+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T13:00:34.322860+0000","last_undegraded":"2024-03-30T13:00:34.322860+0000","last_fullsized":"2024-03-30T13:00:34.322860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:59:32.471198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:32.471198+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:45.757823+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":100,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-03-30T13:00:34.156143+0000","last_change":"2024-03-30T13:00:34.156289+0000","last_active":"2024-03-30T13:00:34.156143+0000","last_peered":"2024-03-30T13:00:34.156143+0000","last_clean":"2024-03-30T13:00:34.156143+0000","last_became_active":"2024-03-30T13:00:14.532540+0000","last_became_peered":"2024-03-30T13:00:14.532540+0000","last_unstale":"2024-03-30T13:00:34.156143+0000","last_undegraded":"2024-03-30T13:00:34.156143+0000","last_fullsized":"2024-03-30T13:00:34.156143+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:00:34.155844+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:54.481678+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":266,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-03-30T13:00:33.013580+0000","last_change":"2024-03-30T12:59:59.675214+0000","last_active":"2024-03-30T13:00:33.013580+0000","last_peered":"2024-03-30T13:00:33.013580+0000","last_clean":"2024-03-30T13:00:33.013580+0000","last_became_active":"2024-03-30T12:58:59.114041+0000","last_became_peered":"2024-03-30T12:58:59.114041+0000","last_unstale":"2024-03-30T13:00:33.013580+0000","last_undegraded":"2024-03-30T13:00:33.013580+0000","last_fullsized":"2024-03-30T13:00:33.013580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:59:59.675050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:59.675050+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:20.917814+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7344560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":315,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-03-30T13:00:34.322923+0000","last_change":"2024-03-30T12:59:55.347262+0000","last_active":"2024-03-30T13:00:34.322923+0000","last_peered":"2024-03-30T13:00:34.322923+0000","last_clean":"2024-03-30T13:00:34.322923+0000","last_became_active":"2024-03-30T12:59:55.305411+0000","last_became_peered":"2024-03-30T12:59:55.305411+0000","last_unstale":"2024-03-30T13:00:34.322923+0000","last_undegraded":"2024-03-30T13:00:34.322923+0000","last_fullsized":"2024-03-30T13:00:34.322923+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:59:18.470423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:18.470423+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:46.794865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":20302592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":23,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-03-30T13:00:34.145434+0000","last_change":"2024-03-30T13:00:14.578743+0000","last_active":"2024-03-30T13:00:34.145434+0000","last_peered":"2024-03-30T13:00:34.145434+0000","last_clean":"2024-03-30T13:00:34.145434+0000","last_became_active":"2024-03-30T13:00:14.533116+0000","last_became_peered":"2024-03-30T13:00:14.533116+0000","last_unstale":"2024-03-30T13:00:34.145434+0000","last_undegraded":"2024-03-30T13:00:34.145434+0000","last_fullsized":"2024-03-30T13:00:34.145434+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T12:59:42.816452+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:01.558414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":318,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-03-30T13:00:33.012855+0000","last_change":"2024-03-30T13:00:14.653924+0000","last_active":"2024-03-30T13:00:33.012855+0000","last_peered":"2024-03-30T13:00:33.012855+0000","last_clean":"2024-03-30T13:00:33.012855+0000","last_became_active":"2024-03-30T13:00:14.539239+0000","last_became_peered":"2024-03-30T13:00:14.539239+0000","last_unstale":"2024-03-30T13:00:33.012855+0000","last_undegraded":"2024-03-30T13:00:33.012855+0000","last_fullsized":"2024-03-30T13:00:33.012855+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:59:37.603454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:37.603454+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:04.259987+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25479082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":285,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-03-30T13:00:33.012990+0000","last_change":"2024-03-30T12:59:28.587515+0000","last_active":"2024-03-30T13:00:33.012990+0000","last_peered":"2024-03-30T13:00:33.012990+0000","last_clean":"2024-03-30T13:00:33.012990+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T13:00:33.012990+0000","last_undegraded":"2024-03-30T13:00:33.012990+0000","last_fullsized":"2024-03-30T13:00:33.012990+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T12:59:28.587085+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T12:59:28.587085+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:52.894911+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":271,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-30T13:00:24.992062+0000","last_change":"2024-03-30T12:59:26.889088+0000","last_active":"2024-03-30T13:00:24.992062+0000","last_peered":"2024-03-30T13:00:24.992062+0000","last_clean":"2024-03-30T13:00:24.992062+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T13:00:24.992062+0000","last_undegraded":"2024-03-30T13:00:24.992062+0000","last_fullsized":"2024-03-30T13:00:24.992062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:59:26.888729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:26.888729+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:49.934982+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":55,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-03-30T13:00:34.134925+0000","last_change":"2024-03-30T12:59:55.354419+0000","last_active":"2024-03-30T13:00:34.134925+0000","last_peered":"2024-03-30T13:00:34.134925+0000","last_clean":"2024-03-30T13:00:34.134925+0000","last_became_active":"2024-03-30T12:59:55.310542+0000","last_became_peered":"2024-03-30T12:59:55.310542+0000","last_unstale":"2024-03-30T13:00:34.134925+0000","last_undegraded":"2024-03-30T13:00:34.134925+0000","last_fullsized":"2024-03-30T13:00:34.134925+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T12:59:24.867416+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:34.375550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":271,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-03-30T13:00:24.992109+0000","last_change":"2024-03-30T12:59:13.956465+0000","last_active":"2024-03-30T13:00:24.992109+0000","last_peered":"2024-03-30T13:00:24.992109+0000","last_clean":"2024-03-30T13:00:24.992109+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T13:00:24.992109+0000","last_undegraded":"2024-03-30T13:00:24.992109+0000","last_fullsized":"2024-03-30T13:00:24.992109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T12:59:13.956186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:13.956186+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:35.452148+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":256,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-03-30T13:00:33.013945+0000","last_change":"2024-03-30T13:00:06.700929+0000","last_active":"2024-03-30T13:00:33.013945+0000","last_peered":"2024-03-30T13:00:33.013945+0000","last_clean":"2024-03-30T13:00:33.013945+0000","last_became_active":"2024-03-30T12:58:59.113215+0000","last_became_peered":"2024-03-30T12:58:59.113215+0000","last_unstale":"2024-03-30T13:00:33.013945+0000","last_undegraded":"2024-03-30T13:00:33.013945+0000","last_fullsized":"2024-03-30T13:00:33.013945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:00:06.700643+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:58:34.875943+0000","last_clean_scrub_stamp":"2024-03-30T13:00:06.700643+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:25.056072+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":121445765,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":107565056,"data_stored":107565056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509503,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43652,"kb_used_data":16632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94328188,"statfs":{"total":96636764160,"available":96592064512,"internally_reserved":0,"allocated":17031168,"data_stored":16155473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542207,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47168,"kb_used_data":20144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94324672,"statfs":{"total":96636764160,"available":96588464128,"internally_reserved":0,"allocated":20627456,"data_stored":19758520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509504,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45784,"kb_used_data":14604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326056,"statfs":{"total":96636764160,"available":96589881344,"internally_reserved":0,"allocated":14954496,"data_stored":14078801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285590,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35316,"kb_used_data":7784,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94336524,"statfs":{"total":96636764160,"available":96600600576,"internally_reserved":0,"allocated":7970816,"data_stored":7095121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":28176082},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509504,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42548,"kb_used_data":15528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329292,"statfs":{"total":96636764160,"available":96593195008,"internally_reserved":0,"allocated":15900672,"data_stored":15024977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":107,"seq":459561500684,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43708,"kb_used_data":11568,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94328132,"statfs":{"total":96636764160,"available":96592007168,"internally_reserved":0,"allocated":11845632,"data_stored":10969937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6042,"internal_metadata":32893030},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574913,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44904,"kb_used_data":17884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94326936,"statfs":{"total":96636764160,"available":96590782464,"internally_reserved":0,"allocated":18313216,"data_stored":17437521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14340096,"data_stored":14340096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6410240,"data_stored":6410240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13393920,"data_stored":13393920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16752640,"data_stored":16752640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19079168,"data_stored":19079168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:00:39.211 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:00:39.211 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:00:39.211 INFO:tasks.thrashosds.thrasher:Reviving osd 4 2024-03-30T13:00:39.211 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-30T13:00:39.212 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-30T13:00:39.222 INFO:tasks.ceph.osd.4:Started 2024-03-30T13:00:39.223 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-30T13:00:39.290 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:00:39.284+0000 7feadf6c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:39.297 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:00:39.292+0000 7feadf6c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:39.297 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:00:39.292+0000 7feadf6c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:39.347 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T13:00:39.347 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T13:00:39.347 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T13:00:39.347 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T13:00:39.347 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T13:00:39.347 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T13:00:39.347 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-03-30T13:00:39.347 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-03-30T13:00:39.347 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-03-30T13:00:39.347 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-03-30T13:00:39.348 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-03-30T13:00:39.348 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T13:00:39.349 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:00:39.349 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-03-30T13:00:39.829 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:00:39.824+0000 7feadf6c87c0 -1 Falling back to public interface 2024-03-30T13:00:43.559 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:00:43.552+0000 7feadf6c87c0 -1 osd.4 125 log_to_monitors true 2024-03-30T13:00:44.116 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:00:44.112+0000 7fead749d640 -1 osd.4 125 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:00:44.350 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-30T13:00:44.485 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T13:00:44.485 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T13:00:44.485 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T13:00:44.485 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T13:00:44.499 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:00:44.499 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-03-30T13:00:44.499 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 4 2024-03-30T13:00:44.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:44.625+0000 7f90895d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:44.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:44.641+0000 7f90895d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:44.823 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:00:44.821+0000 7fbb38f08640 -1 mon.a@0(leader).osd e134 definitely_dead 0 2024-03-30T13:00:45.098 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.4. 2024-03-30T13:00:45.124 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-03-30T13:00:45.124 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:00:45.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:45.237+0000 7fa3e6d41640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:45.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:45.253+0000 7fa3e6d41640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:46.094 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:00:46.126 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:00:46.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:46.237+0000 7fdd5d145640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:46.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:46.253+0000 7fdd5d145640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:47.123 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:00:47.147 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:00:47.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:47.261+0000 7f15b6c58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:47.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:47.277+0000 7f15b6c58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:48.137 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:00:48.162 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:00:48.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:48.277+0000 7fe03edc3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:48.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:48.293+0000 7fe03edc3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:49.140 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:00:49.166 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:00:49.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:49.285+0000 7f4dd5194640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:49.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:49.301+0000 7f4dd5194640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:50.150 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:00:50.172 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:00:50.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:50.277+0000 7f712c4d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:50.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:50.293+0000 7f712c4d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:51.159 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:00:51.185 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:00:51.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:51.301+0000 7f31fc830640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:51.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:51.317+0000 7f31fc830640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:52.169 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:00:52.193 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:00:52.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:52.309+0000 7f7d6dc60640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:52.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:52.325+0000 7f7d6dc60640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:53.173 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:00:53.201 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:00:53.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:53.317+0000 7faea6def640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:53.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:53.333+0000 7faea6def640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:54.188 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:00:54.212 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:00:54.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:54.329+0000 7fdf9d0bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:54.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:54.345+0000 7fdf9d0bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:55.199 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:00:55.223 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:00:55.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:55.341+0000 7f5995c03640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:55.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:55.357+0000 7f5995c03640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:56.211 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:00:56.239 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:00:56.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:56.357+0000 7f15d072c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:56.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:56.373+0000 7f15d072c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:57.231 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:00:57.259 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:00:57.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:57.377+0000 7f65964a0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:57.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:57.397+0000 7f65964a0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:58.228 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:00:58.253 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:00:58.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:58.373+0000 7f56e3762640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:58.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:58.389+0000 7f56e3762640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:59.244 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:00:59.271 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:00:59.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:59.385+0000 7fdfa29a1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:00:59.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:00:59.401+0000 7fdfa29a1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:00.253 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:01:00.276 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:01:00.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:00.393+0000 7f150da76640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:00.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:00.409+0000 7f150da76640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:01.264 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:01:01.289 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:01:01.289 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:01:01.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:01.405+0000 7f3c55cc4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:01.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:01.417+0000 7f3c55cc4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:01.595 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:01:01.595 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:01:01.613 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2024-03-30T13:01:01.265132+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":121445765,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":337696,"kb_used_data":106608,"kb_used_omap":23,"kb_used_meta":230952,"kb_avail":754637024,"statfs":{"total":773094113280,"available":772748312576,"internally_reserved":0,"allocated":109166592,"data_stored":101276744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24463,"internal_metadata":236494961},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.040480"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":328,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-30T13:00:58.231785+0000","last_change":"2024-03-30T13:00:46.110671+0000","last_active":"2024-03-30T13:00:58.231785+0000","last_peered":"2024-03-30T13:00:58.231785+0000","last_clean":"2024-03-30T13:00:58.231785+0000","last_became_active":"2024-03-30T13:00:46.109346+0000","last_became_peered":"2024-03-30T13:00:46.109346+0000","last_unstale":"2024-03-30T13:00:58.231785+0000","last_undegraded":"2024-03-30T13:00:58.231785+0000","last_fullsized":"2024-03-30T13:00:58.231785+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:59:56.672842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:56.672842+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:08.297620+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":291,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-30T13:00:59.247512+0000","last_change":"2024-03-30T13:00:58.359297+0000","last_active":"2024-03-30T13:00:59.247512+0000","last_peered":"2024-03-30T13:00:59.247512+0000","last_clean":"2024-03-30T13:00:59.247512+0000","last_became_active":"2024-03-30T13:00:46.110558+0000","last_became_peered":"2024-03-30T13:00:46.110558+0000","last_unstale":"2024-03-30T13:00:59.247512+0000","last_undegraded":"2024-03-30T13:00:59.247512+0000","last_fullsized":"2024-03-30T13:00:59.247512+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:00:58.359040+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:00:58.359040+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:20.772622+0000","scrub_duration":927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":109,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-30T13:00:56.644530+0000","last_change":"2024-03-30T13:00:46.115546+0000","last_active":"2024-03-30T13:00:56.644530+0000","last_peered":"2024-03-30T13:00:56.644530+0000","last_clean":"2024-03-30T13:00:56.644530+0000","last_became_active":"2024-03-30T13:00:46.112427+0000","last_became_peered":"2024-03-30T13:00:46.112427+0000","last_unstale":"2024-03-30T13:00:56.644530+0000","last_undegraded":"2024-03-30T13:00:56.644530+0000","last_fullsized":"2024-03-30T13:00:56.644530+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:59:47.146312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:47.146312+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:59.864920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":95,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-30T13:00:56.644451+0000","last_change":"2024-03-30T13:00:35.159895+0000","last_active":"2024-03-30T13:00:56.644451+0000","last_peered":"2024-03-30T13:00:56.644451+0000","last_clean":"2024-03-30T13:00:56.644451+0000","last_became_active":"2024-03-30T12:59:55.316713+0000","last_became_peered":"2024-03-30T12:59:55.316713+0000","last_unstale":"2024-03-30T13:00:56.644451+0000","last_undegraded":"2024-03-30T13:00:56.644451+0000","last_fullsized":"2024-03-30T13:00:56.644451+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:00:35.159640+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:00:35.159640+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:04.854390+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":294,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-30T13:00:58.234115+0000","last_change":"2024-03-30T13:00:01.952302+0000","last_active":"2024-03-30T13:00:58.234115+0000","last_peered":"2024-03-30T13:00:58.234115+0000","last_clean":"2024-03-30T13:00:58.234115+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T13:00:58.234115+0000","last_undegraded":"2024-03-30T13:00:58.234115+0000","last_fullsized":"2024-03-30T13:00:58.234115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:00:01.951942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:01.951942+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:08.852178+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":311,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-30T13:00:56.643822+0000","last_change":"2024-03-30T13:00:46.414441+0000","last_active":"2024-03-30T13:00:56.643822+0000","last_peered":"2024-03-30T13:00:56.643822+0000","last_clean":"2024-03-30T13:00:56.643822+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T13:00:56.643822+0000","last_undegraded":"2024-03-30T13:00:56.643822+0000","last_fullsized":"2024-03-30T13:00:56.643822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:00:46.414102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:46.414102+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:05.949447+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":136,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-30T13:00:56.644503+0000","last_change":"2024-03-30T13:00:46.115096+0000","last_active":"2024-03-30T13:00:56.644503+0000","last_peered":"2024-03-30T13:00:56.644503+0000","last_clean":"2024-03-30T13:00:56.644503+0000","last_became_active":"2024-03-30T13:00:46.114131+0000","last_became_peered":"2024-03-30T13:00:46.114131+0000","last_unstale":"2024-03-30T13:00:56.644503+0000","last_undegraded":"2024-03-30T13:00:56.644503+0000","last_fullsized":"2024-03-30T13:00:56.644503+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:00:34.155844+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:47.134436+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":298,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-30T13:00:57.227588+0000","last_change":"2024-03-30T12:59:59.675214+0000","last_active":"2024-03-30T13:00:57.227588+0000","last_peered":"2024-03-30T13:00:57.227588+0000","last_clean":"2024-03-30T13:00:57.227588+0000","last_became_active":"2024-03-30T12:58:59.114041+0000","last_became_peered":"2024-03-30T12:58:59.114041+0000","last_unstale":"2024-03-30T13:00:57.227588+0000","last_undegraded":"2024-03-30T13:00:57.227588+0000","last_fullsized":"2024-03-30T13:00:57.227588+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:59:59.675050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:59.675050+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:20.917814+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7344560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":352,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-30T13:00:56.644032+0000","last_change":"2024-03-30T13:00:49.392191+0000","last_active":"2024-03-30T13:00:56.644032+0000","last_peered":"2024-03-30T13:00:56.644032+0000","last_clean":"2024-03-30T13:00:56.644032+0000","last_became_active":"2024-03-30T12:59:55.305411+0000","last_became_peered":"2024-03-30T12:59:55.305411+0000","last_unstale":"2024-03-30T13:00:56.644032+0000","last_undegraded":"2024-03-30T13:00:56.644032+0000","last_fullsized":"2024-03-30T13:00:56.644032+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:00:49.391955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:49.391955+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:04.091733+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":20302592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":219,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-30T13:00:59.247404+0000","last_change":"2024-03-30T13:00:46.110349+0000","last_active":"2024-03-30T13:00:59.247404+0000","last_peered":"2024-03-30T13:00:59.247404+0000","last_clean":"2024-03-30T13:00:59.247404+0000","last_became_active":"2024-03-30T13:00:46.109054+0000","last_became_peered":"2024-03-30T13:00:46.109054+0000","last_unstale":"2024-03-30T13:00:59.247404+0000","last_undegraded":"2024-03-30T13:00:59.247404+0000","last_fullsized":"2024-03-30T13:00:59.247404+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T12:59:42.816452+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:12.073828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":355,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-30T13:00:58.231773+0000","last_change":"2024-03-30T13:00:46.107422+0000","last_active":"2024-03-30T13:00:58.231773+0000","last_peered":"2024-03-30T13:00:58.231773+0000","last_clean":"2024-03-30T13:00:58.231773+0000","last_became_active":"2024-03-30T13:00:46.106211+0000","last_became_peered":"2024-03-30T13:00:46.106211+0000","last_unstale":"2024-03-30T13:00:58.231773+0000","last_undegraded":"2024-03-30T13:00:58.231773+0000","last_fullsized":"2024-03-30T13:00:58.231773+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:59:37.603454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:37.603454+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:03.418078+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25479082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":324,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-30T13:00:58.231919+0000","last_change":"2024-03-30T13:00:53.751657+0000","last_active":"2024-03-30T13:00:58.231919+0000","last_peered":"2024-03-30T13:00:58.231919+0000","last_clean":"2024-03-30T13:00:58.231919+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T13:00:58.231919+0000","last_undegraded":"2024-03-30T13:00:58.231919+0000","last_fullsized":"2024-03-30T13:00:58.231919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:00:53.751391+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T13:00:53.751391+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:04.418173+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":314,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-30T13:00:58.234087+0000","last_change":"2024-03-30T13:00:50.897620+0000","last_active":"2024-03-30T13:00:58.234087+0000","last_peered":"2024-03-30T13:00:58.234087+0000","last_clean":"2024-03-30T13:00:58.234087+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T13:00:58.234087+0000","last_undegraded":"2024-03-30T13:00:58.234087+0000","last_fullsized":"2024-03-30T13:00:58.234087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:00:50.897208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:50.897208+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:00.526987+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":93,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-30T13:00:56.644475+0000","last_change":"2024-03-30T13:00:39.088475+0000","last_active":"2024-03-30T13:00:56.644475+0000","last_peered":"2024-03-30T13:00:56.644475+0000","last_clean":"2024-03-30T13:00:56.644475+0000","last_became_active":"2024-03-30T12:59:55.310542+0000","last_became_peered":"2024-03-30T12:59:55.310542+0000","last_unstale":"2024-03-30T13:00:56.644475+0000","last_undegraded":"2024-03-30T13:00:56.644475+0000","last_fullsized":"2024-03-30T13:00:56.644475+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:00:39.088323+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:00:39.088323+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:05.618294+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":314,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-30T13:00:58.234238+0000","last_change":"2024-03-30T13:00:36.953929+0000","last_active":"2024-03-30T13:00:58.234238+0000","last_peered":"2024-03-30T13:00:58.234238+0000","last_clean":"2024-03-30T13:00:58.234238+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T13:00:58.234238+0000","last_undegraded":"2024-03-30T13:00:58.234238+0000","last_fullsized":"2024-03-30T13:00:58.234238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:00:36.953660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:36.953660+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:48.382597+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":288,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-30T13:00:57.228003+0000","last_change":"2024-03-30T13:00:06.700929+0000","last_active":"2024-03-30T13:00:57.228003+0000","last_peered":"2024-03-30T13:00:57.228003+0000","last_clean":"2024-03-30T13:00:57.228003+0000","last_became_active":"2024-03-30T12:58:59.113215+0000","last_became_peered":"2024-03-30T12:58:59.113215+0000","last_unstale":"2024-03-30T13:00:57.228003+0000","last_undegraded":"2024-03-30T13:00:57.228003+0000","last_fullsized":"2024-03-30T13:00:57.228003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:00:06.700643+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:58:34.875943+0000","last_clean_scrub_stamp":"2024-03-30T13:00:06.700643+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:25.056072+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":121445765,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":95059968,"data_stored":95059968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509508,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43856,"kb_used_data":16836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94327984,"statfs":{"total":96636764160,"available":96591855616,"internally_reserved":0,"allocated":17240064,"data_stored":16250445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542212,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47380,"kb_used_data":20360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94324460,"statfs":{"total":96636764160,"available":96588247040,"internally_reserved":0,"allocated":20848640,"data_stored":19859021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":135,"seq":579820584965,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45436,"kb_used_data":13296,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94326404,"statfs":{"total":96636764160,"available":96590237696,"internally_reserved":0,"allocated":13615104,"data_stored":12618734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6047,"internal_metadata":32893025},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509509,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39244,"kb_used_data":8064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332596,"statfs":{"total":96636764160,"available":96596578304,"internally_reserved":0,"allocated":8257536,"data_stored":7261166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285594,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34416,"kb_used_data":6884,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94337424,"statfs":{"total":96636764160,"available":96601522176,"internally_reserved":0,"allocated":7049216,"data_stored":6066348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":28176082},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509509,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42248,"kb_used_data":15216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329592,"statfs":{"total":96636764160,"available":96593502208,"internally_reserved":0,"allocated":15581184,"data_stored":14598316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":107,"seq":459561500688,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43888,"kb_used_data":11748,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94327952,"statfs":{"total":96636764160,"available":96591822848,"internally_reserved":0,"allocated":12029952,"data_stored":11053885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6042,"internal_metadata":32893030},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574917,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41228,"kb_used_data":14204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330612,"statfs":{"total":96636764160,"available":96594546688,"internally_reserved":0,"allocated":14544896,"data_stored":13568829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19079168,"data_stored":19079168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:01:01.614 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:01:01.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:01.729+0000 7f5214157640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:01.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:01.745+0000 7f5214157640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:01.919 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:01:01.919 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:01:01.937 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-03-30T13:01:01.875646+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":121445765,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":337696,"kb_used_data":106608,"kb_used_omap":23,"kb_used_meta":230952,"kb_avail":754637024,"statfs":{"total":773094113280,"available":772748312576,"internally_reserved":0,"allocated":109166592,"data_stored":101276744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24463,"internal_metadata":236494961},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001101"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":328,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-30T13:00:58.231785+0000","last_change":"2024-03-30T13:00:46.110671+0000","last_active":"2024-03-30T13:00:58.231785+0000","last_peered":"2024-03-30T13:00:58.231785+0000","last_clean":"2024-03-30T13:00:58.231785+0000","last_became_active":"2024-03-30T13:00:46.109346+0000","last_became_peered":"2024-03-30T13:00:46.109346+0000","last_unstale":"2024-03-30T13:00:58.231785+0000","last_undegraded":"2024-03-30T13:00:58.231785+0000","last_fullsized":"2024-03-30T13:00:58.231785+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:59:56.672842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:56.672842+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:08.297620+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":291,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-30T13:00:59.247512+0000","last_change":"2024-03-30T13:00:58.359297+0000","last_active":"2024-03-30T13:00:59.247512+0000","last_peered":"2024-03-30T13:00:59.247512+0000","last_clean":"2024-03-30T13:00:59.247512+0000","last_became_active":"2024-03-30T13:00:46.110558+0000","last_became_peered":"2024-03-30T13:00:46.110558+0000","last_unstale":"2024-03-30T13:00:59.247512+0000","last_undegraded":"2024-03-30T13:00:59.247512+0000","last_fullsized":"2024-03-30T13:00:59.247512+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:00:58.359040+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:00:58.359040+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:20.772622+0000","scrub_duration":927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":109,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-30T13:00:56.644530+0000","last_change":"2024-03-30T13:00:46.115546+0000","last_active":"2024-03-30T13:00:56.644530+0000","last_peered":"2024-03-30T13:00:56.644530+0000","last_clean":"2024-03-30T13:00:56.644530+0000","last_became_active":"2024-03-30T13:00:46.112427+0000","last_became_peered":"2024-03-30T13:00:46.112427+0000","last_unstale":"2024-03-30T13:00:56.644530+0000","last_undegraded":"2024-03-30T13:00:56.644530+0000","last_fullsized":"2024-03-30T13:00:56.644530+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:59:47.146312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:47.146312+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:59.864920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":95,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-30T13:00:56.644451+0000","last_change":"2024-03-30T13:00:35.159895+0000","last_active":"2024-03-30T13:00:56.644451+0000","last_peered":"2024-03-30T13:00:56.644451+0000","last_clean":"2024-03-30T13:00:56.644451+0000","last_became_active":"2024-03-30T12:59:55.316713+0000","last_became_peered":"2024-03-30T12:59:55.316713+0000","last_unstale":"2024-03-30T13:00:56.644451+0000","last_undegraded":"2024-03-30T13:00:56.644451+0000","last_fullsized":"2024-03-30T13:00:56.644451+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:00:35.159640+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:00:35.159640+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:04.854390+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":294,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-30T13:00:58.234115+0000","last_change":"2024-03-30T13:00:01.952302+0000","last_active":"2024-03-30T13:00:58.234115+0000","last_peered":"2024-03-30T13:00:58.234115+0000","last_clean":"2024-03-30T13:00:58.234115+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T13:00:58.234115+0000","last_undegraded":"2024-03-30T13:00:58.234115+0000","last_fullsized":"2024-03-30T13:00:58.234115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:00:01.951942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:01.951942+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:08.852178+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":311,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-30T13:00:56.643822+0000","last_change":"2024-03-30T13:00:46.414441+0000","last_active":"2024-03-30T13:00:56.643822+0000","last_peered":"2024-03-30T13:00:56.643822+0000","last_clean":"2024-03-30T13:00:56.643822+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T13:00:56.643822+0000","last_undegraded":"2024-03-30T13:00:56.643822+0000","last_fullsized":"2024-03-30T13:00:56.643822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:00:46.414102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:46.414102+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:05.949447+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":136,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-30T13:00:56.644503+0000","last_change":"2024-03-30T13:00:46.115096+0000","last_active":"2024-03-30T13:00:56.644503+0000","last_peered":"2024-03-30T13:00:56.644503+0000","last_clean":"2024-03-30T13:00:56.644503+0000","last_became_active":"2024-03-30T13:00:46.114131+0000","last_became_peered":"2024-03-30T13:00:46.114131+0000","last_unstale":"2024-03-30T13:00:56.644503+0000","last_undegraded":"2024-03-30T13:00:56.644503+0000","last_fullsized":"2024-03-30T13:00:56.644503+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:00:34.155844+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:47.134436+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":298,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-30T13:00:57.227588+0000","last_change":"2024-03-30T12:59:59.675214+0000","last_active":"2024-03-30T13:00:57.227588+0000","last_peered":"2024-03-30T13:00:57.227588+0000","last_clean":"2024-03-30T13:00:57.227588+0000","last_became_active":"2024-03-30T12:58:59.114041+0000","last_became_peered":"2024-03-30T12:58:59.114041+0000","last_unstale":"2024-03-30T13:00:57.227588+0000","last_undegraded":"2024-03-30T13:00:57.227588+0000","last_fullsized":"2024-03-30T13:00:57.227588+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:59:59.675050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:59.675050+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:20.917814+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7344560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":352,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-30T13:00:56.644032+0000","last_change":"2024-03-30T13:00:49.392191+0000","last_active":"2024-03-30T13:00:56.644032+0000","last_peered":"2024-03-30T13:00:56.644032+0000","last_clean":"2024-03-30T13:00:56.644032+0000","last_became_active":"2024-03-30T12:59:55.305411+0000","last_became_peered":"2024-03-30T12:59:55.305411+0000","last_unstale":"2024-03-30T13:00:56.644032+0000","last_undegraded":"2024-03-30T13:00:56.644032+0000","last_fullsized":"2024-03-30T13:00:56.644032+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:00:49.391955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:49.391955+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:04.091733+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":20302592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":219,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-30T13:00:59.247404+0000","last_change":"2024-03-30T13:00:46.110349+0000","last_active":"2024-03-30T13:00:59.247404+0000","last_peered":"2024-03-30T13:00:59.247404+0000","last_clean":"2024-03-30T13:00:59.247404+0000","last_became_active":"2024-03-30T13:00:46.109054+0000","last_became_peered":"2024-03-30T13:00:46.109054+0000","last_unstale":"2024-03-30T13:00:59.247404+0000","last_undegraded":"2024-03-30T13:00:59.247404+0000","last_fullsized":"2024-03-30T13:00:59.247404+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T12:59:42.816452+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:12.073828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":355,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-30T13:00:58.231773+0000","last_change":"2024-03-30T13:00:46.107422+0000","last_active":"2024-03-30T13:00:58.231773+0000","last_peered":"2024-03-30T13:00:58.231773+0000","last_clean":"2024-03-30T13:00:58.231773+0000","last_became_active":"2024-03-30T13:00:46.106211+0000","last_became_peered":"2024-03-30T13:00:46.106211+0000","last_unstale":"2024-03-30T13:00:58.231773+0000","last_undegraded":"2024-03-30T13:00:58.231773+0000","last_fullsized":"2024-03-30T13:00:58.231773+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:59:37.603454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:37.603454+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:03.418078+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25479082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":324,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-30T13:00:58.231919+0000","last_change":"2024-03-30T13:00:53.751657+0000","last_active":"2024-03-30T13:00:58.231919+0000","last_peered":"2024-03-30T13:00:58.231919+0000","last_clean":"2024-03-30T13:00:58.231919+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T13:00:58.231919+0000","last_undegraded":"2024-03-30T13:00:58.231919+0000","last_fullsized":"2024-03-30T13:00:58.231919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:00:53.751391+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T13:00:53.751391+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:04.418173+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":314,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-30T13:00:58.234087+0000","last_change":"2024-03-30T13:00:50.897620+0000","last_active":"2024-03-30T13:00:58.234087+0000","last_peered":"2024-03-30T13:00:58.234087+0000","last_clean":"2024-03-30T13:00:58.234087+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T13:00:58.234087+0000","last_undegraded":"2024-03-30T13:00:58.234087+0000","last_fullsized":"2024-03-30T13:00:58.234087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:00:50.897208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:50.897208+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:00.526987+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":93,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-30T13:00:56.644475+0000","last_change":"2024-03-30T13:00:39.088475+0000","last_active":"2024-03-30T13:00:56.644475+0000","last_peered":"2024-03-30T13:00:56.644475+0000","last_clean":"2024-03-30T13:00:56.644475+0000","last_became_active":"2024-03-30T12:59:55.310542+0000","last_became_peered":"2024-03-30T12:59:55.310542+0000","last_unstale":"2024-03-30T13:00:56.644475+0000","last_undegraded":"2024-03-30T13:00:56.644475+0000","last_fullsized":"2024-03-30T13:00:56.644475+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:00:39.088323+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:00:39.088323+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:05.618294+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":314,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-30T13:00:58.234238+0000","last_change":"2024-03-30T13:00:36.953929+0000","last_active":"2024-03-30T13:00:58.234238+0000","last_peered":"2024-03-30T13:00:58.234238+0000","last_clean":"2024-03-30T13:00:58.234238+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T13:00:58.234238+0000","last_undegraded":"2024-03-30T13:00:58.234238+0000","last_fullsized":"2024-03-30T13:00:58.234238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:00:36.953660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:36.953660+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:48.382597+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":288,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-30T13:00:57.228003+0000","last_change":"2024-03-30T13:00:06.700929+0000","last_active":"2024-03-30T13:00:57.228003+0000","last_peered":"2024-03-30T13:00:57.228003+0000","last_clean":"2024-03-30T13:00:57.228003+0000","last_became_active":"2024-03-30T12:58:59.113215+0000","last_became_peered":"2024-03-30T12:58:59.113215+0000","last_unstale":"2024-03-30T13:00:57.228003+0000","last_undegraded":"2024-03-30T13:00:57.228003+0000","last_fullsized":"2024-03-30T13:00:57.228003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:00:06.700643+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:58:34.875943+0000","last_clean_scrub_stamp":"2024-03-30T13:00:06.700643+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:25.056072+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":121445765,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":95059968,"data_stored":95059968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509508,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43856,"kb_used_data":16836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94327984,"statfs":{"total":96636764160,"available":96591855616,"internally_reserved":0,"allocated":17240064,"data_stored":16250445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542212,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47380,"kb_used_data":20360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94324460,"statfs":{"total":96636764160,"available":96588247040,"internally_reserved":0,"allocated":20848640,"data_stored":19859021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":135,"seq":579820584965,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45436,"kb_used_data":13296,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94326404,"statfs":{"total":96636764160,"available":96590237696,"internally_reserved":0,"allocated":13615104,"data_stored":12618734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6047,"internal_metadata":32893025},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509509,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39244,"kb_used_data":8064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332596,"statfs":{"total":96636764160,"available":96596578304,"internally_reserved":0,"allocated":8257536,"data_stored":7261166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285594,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34416,"kb_used_data":6884,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94337424,"statfs":{"total":96636764160,"available":96601522176,"internally_reserved":0,"allocated":7049216,"data_stored":6066348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":28176082},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509509,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42248,"kb_used_data":15216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329592,"statfs":{"total":96636764160,"available":96593502208,"internally_reserved":0,"allocated":15581184,"data_stored":14598316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":107,"seq":459561500688,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43888,"kb_used_data":11748,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94327952,"statfs":{"total":96636764160,"available":96591822848,"internally_reserved":0,"allocated":12029952,"data_stored":11053885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6042,"internal_metadata":32893030},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574917,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41228,"kb_used_data":14204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330612,"statfs":{"total":96636764160,"available":96594546688,"internally_reserved":0,"allocated":14544896,"data_stored":13568829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19079168,"data_stored":19079168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:01:01.938 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:01:01.938 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:01:02.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:02.041+0000 7f48f58ee640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:02.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:02.053+0000 7f48f58ee640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:02.228 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:01:02.229 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":151,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:01:01.255199+0000","last_up_change":"2024-03-30T13:00:45.089072+0000","last_in_change":"2024-03-30T13:00:45.089072+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":11,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":77,"up_from":83,"up_thru":102,"down_at":78,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6818","nonce":3104224893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6821","nonce":3104224893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6828","nonce":3104224893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6825","nonce":3104224893}]},"public_addr":"172.21.15.67:6818/3104224893","cluster_addr":"172.21.15.67:6821/3104224893","heartbeat_back_addr":"172.21.15.67:6828/3104224893","heartbeat_front_addr":"172.21.15.67:6825/3104224893","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6801","nonce":1504348724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6803","nonce":1504348724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6807","nonce":1504348724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6805","nonce":1504348724}]},"public_addr":"172.21.15.67:6801/1504348724","cluster_addr":"172.21.15.67:6803/1504348724","heartbeat_back_addr":"172.21.15.67:6807/1504348724","heartbeat_front_addr":"172.21.15.67:6805/1504348724","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":36,"last_clean_end":125,"up_from":135,"up_thru":135,"down_at":126,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":2818291404},{"type":"v1","addr":"172.21.15.82:6803","nonce":2818291404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":2818291404},{"type":"v1","addr":"172.21.15.82:6807","nonce":2818291404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":2818291404},{"type":"v1","addr":"172.21.15.82:6815","nonce":2818291404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":2818291404},{"type":"v1","addr":"172.21.15.82:6811","nonce":2818291404}]},"public_addr":"172.21.15.82:6803/2818291404","cluster_addr":"172.21.15.82:6807/2818291404","heartbeat_back_addr":"172.21.15.82:6815/2818291404","heartbeat_front_addr":"172.21.15.82:6811/2818291404","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":102,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6817","nonce":3467856022}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6819","nonce":3467856022}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6823","nonce":3467856022}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":3467856022},{"type":"v1","addr":"172.21.15.82:6821","nonce":3467856022}]},"public_addr":"172.21.15.82:6817/3467856022","cluster_addr":"172.21.15.82:6819/3467856022","heartbeat_back_addr":"172.21.15.82:6823/3467856022","heartbeat_front_addr":"172.21.15.82:6821/3467856022","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6801","nonce":968508548}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6806","nonce":968508548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6814","nonce":968508548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6810","nonce":968508548}]},"public_addr":"172.21.15.82:6801/968508548","cluster_addr":"172.21.15.82:6806/968508548","heartbeat_back_addr":"172.21.15.82:6814/968508548","heartbeat_front_addr":"172.21.15.82:6810/968508548","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":100,"up_from":107,"up_thru":135,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2568135668},{"type":"v1","addr":"172.21.15.82:6825","nonce":2568135668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2568135668},{"type":"v1","addr":"172.21.15.82:6827","nonce":2568135668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2568135668},{"type":"v1","addr":"172.21.15.82:6831","nonce":2568135668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2568135668},{"type":"v1","addr":"172.21.15.82:6829","nonce":2568135668}]},"public_addr":"172.21.15.82:6825/2568135668","cluster_addr":"172.21.15.82:6827/2568135668","heartbeat_back_addr":"172.21.15.82:6831/2568135668","heartbeat_front_addr":"172.21.15.82:6829/2568135668","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:00:38.871141+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:00:32.475206+0000","dead_epoch":78},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:00:24.547931+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:00:40.637135+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:00:58.601253+0000","dead_epoch":126},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:00:21.973060+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:00:20.015587+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:00:58.181517+0000","dead_epoch":101}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:01:02.247 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:01:02.247 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:01:02.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:02.361+0000 7f0307294640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:02.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:02.377+0000 7f0307294640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:02.552 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:01:02.571 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:01:02.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:02.681+0000 7fde88e61640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:02.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:02.697+0000 7fde88e61640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:02.877 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:01:02.877 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:01:02.896 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:01:02.896 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:01:02.896 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:01:02.896 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:01:02.896 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 3, 5, 6, 1, 7, 4] 2024-03-30T13:01:02.896 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:01:02.896 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:01:03.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:03.001+0000 7f555dda1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:03.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:03.013+0000 7f555dda1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:03.184 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:01:03.185 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:01:03.201 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-03-30T13:01:01.875646+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":121445765,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":337696,"kb_used_data":106608,"kb_used_omap":23,"kb_used_meta":230952,"kb_avail":754637024,"statfs":{"total":773094113280,"available":772748312576,"internally_reserved":0,"allocated":109166592,"data_stored":101276744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24463,"internal_metadata":236494961},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001101"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":328,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-30T13:00:58.231785+0000","last_change":"2024-03-30T13:00:46.110671+0000","last_active":"2024-03-30T13:00:58.231785+0000","last_peered":"2024-03-30T13:00:58.231785+0000","last_clean":"2024-03-30T13:00:58.231785+0000","last_became_active":"2024-03-30T13:00:46.109346+0000","last_became_peered":"2024-03-30T13:00:46.109346+0000","last_unstale":"2024-03-30T13:00:58.231785+0000","last_undegraded":"2024-03-30T13:00:58.231785+0000","last_fullsized":"2024-03-30T13:00:58.231785+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:59:56.672842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:56.672842+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:08.297620+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":291,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-30T13:00:59.247512+0000","last_change":"2024-03-30T13:00:58.359297+0000","last_active":"2024-03-30T13:00:59.247512+0000","last_peered":"2024-03-30T13:00:59.247512+0000","last_clean":"2024-03-30T13:00:59.247512+0000","last_became_active":"2024-03-30T13:00:46.110558+0000","last_became_peered":"2024-03-30T13:00:46.110558+0000","last_unstale":"2024-03-30T13:00:59.247512+0000","last_undegraded":"2024-03-30T13:00:59.247512+0000","last_fullsized":"2024-03-30T13:00:59.247512+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:00:58.359040+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:00:58.359040+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:20.772622+0000","scrub_duration":927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":109,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-30T13:00:56.644530+0000","last_change":"2024-03-30T13:00:46.115546+0000","last_active":"2024-03-30T13:00:56.644530+0000","last_peered":"2024-03-30T13:00:56.644530+0000","last_clean":"2024-03-30T13:00:56.644530+0000","last_became_active":"2024-03-30T13:00:46.112427+0000","last_became_peered":"2024-03-30T13:00:46.112427+0000","last_unstale":"2024-03-30T13:00:56.644530+0000","last_undegraded":"2024-03-30T13:00:56.644530+0000","last_fullsized":"2024-03-30T13:00:56.644530+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:59:47.146312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:47.146312+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:00:59.864920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":95,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-30T13:00:56.644451+0000","last_change":"2024-03-30T13:00:35.159895+0000","last_active":"2024-03-30T13:00:56.644451+0000","last_peered":"2024-03-30T13:00:56.644451+0000","last_clean":"2024-03-30T13:00:56.644451+0000","last_became_active":"2024-03-30T12:59:55.316713+0000","last_became_peered":"2024-03-30T12:59:55.316713+0000","last_unstale":"2024-03-30T13:00:56.644451+0000","last_undegraded":"2024-03-30T13:00:56.644451+0000","last_fullsized":"2024-03-30T13:00:56.644451+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:00:35.159640+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:00:35.159640+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:04.854390+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":294,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-30T13:00:58.234115+0000","last_change":"2024-03-30T13:00:01.952302+0000","last_active":"2024-03-30T13:00:58.234115+0000","last_peered":"2024-03-30T13:00:58.234115+0000","last_clean":"2024-03-30T13:00:58.234115+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T13:00:58.234115+0000","last_undegraded":"2024-03-30T13:00:58.234115+0000","last_fullsized":"2024-03-30T13:00:58.234115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:00:01.951942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:01.951942+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:08.852178+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":311,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-30T13:00:56.643822+0000","last_change":"2024-03-30T13:00:46.414441+0000","last_active":"2024-03-30T13:00:56.643822+0000","last_peered":"2024-03-30T13:00:56.643822+0000","last_clean":"2024-03-30T13:00:56.643822+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T13:00:56.643822+0000","last_undegraded":"2024-03-30T13:00:56.643822+0000","last_fullsized":"2024-03-30T13:00:56.643822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:00:46.414102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:46.414102+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:05.949447+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":136,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-30T13:00:56.644503+0000","last_change":"2024-03-30T13:00:46.115096+0000","last_active":"2024-03-30T13:00:56.644503+0000","last_peered":"2024-03-30T13:00:56.644503+0000","last_clean":"2024-03-30T13:00:56.644503+0000","last_became_active":"2024-03-30T13:00:46.114131+0000","last_became_peered":"2024-03-30T13:00:46.114131+0000","last_unstale":"2024-03-30T13:00:56.644503+0000","last_undegraded":"2024-03-30T13:00:56.644503+0000","last_fullsized":"2024-03-30T13:00:56.644503+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:00:34.155844+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:47.134436+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":298,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-30T13:00:57.227588+0000","last_change":"2024-03-30T12:59:59.675214+0000","last_active":"2024-03-30T13:00:57.227588+0000","last_peered":"2024-03-30T13:00:57.227588+0000","last_clean":"2024-03-30T13:00:57.227588+0000","last_became_active":"2024-03-30T12:58:59.114041+0000","last_became_peered":"2024-03-30T12:58:59.114041+0000","last_unstale":"2024-03-30T13:00:57.227588+0000","last_undegraded":"2024-03-30T13:00:57.227588+0000","last_fullsized":"2024-03-30T13:00:57.227588+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:59:59.675050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:59.675050+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:20.917814+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7344560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":352,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-30T13:00:56.644032+0000","last_change":"2024-03-30T13:00:49.392191+0000","last_active":"2024-03-30T13:00:56.644032+0000","last_peered":"2024-03-30T13:00:56.644032+0000","last_clean":"2024-03-30T13:00:56.644032+0000","last_became_active":"2024-03-30T12:59:55.305411+0000","last_became_peered":"2024-03-30T12:59:55.305411+0000","last_unstale":"2024-03-30T13:00:56.644032+0000","last_undegraded":"2024-03-30T13:00:56.644032+0000","last_fullsized":"2024-03-30T13:00:56.644032+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:00:49.391955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:49.391955+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:04.091733+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":20302592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":219,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-30T13:00:59.247404+0000","last_change":"2024-03-30T13:00:46.110349+0000","last_active":"2024-03-30T13:00:59.247404+0000","last_peered":"2024-03-30T13:00:59.247404+0000","last_clean":"2024-03-30T13:00:59.247404+0000","last_became_active":"2024-03-30T13:00:46.109054+0000","last_became_peered":"2024-03-30T13:00:46.109054+0000","last_unstale":"2024-03-30T13:00:59.247404+0000","last_undegraded":"2024-03-30T13:00:59.247404+0000","last_fullsized":"2024-03-30T13:00:59.247404+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T12:59:42.816452+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:12.073828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":355,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-30T13:00:58.231773+0000","last_change":"2024-03-30T13:00:46.107422+0000","last_active":"2024-03-30T13:00:58.231773+0000","last_peered":"2024-03-30T13:00:58.231773+0000","last_clean":"2024-03-30T13:00:58.231773+0000","last_became_active":"2024-03-30T13:00:46.106211+0000","last_became_peered":"2024-03-30T13:00:46.106211+0000","last_unstale":"2024-03-30T13:00:58.231773+0000","last_undegraded":"2024-03-30T13:00:58.231773+0000","last_fullsized":"2024-03-30T13:00:58.231773+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:59:37.603454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:37.603454+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:03.418078+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25479082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":324,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-30T13:00:58.231919+0000","last_change":"2024-03-30T13:00:53.751657+0000","last_active":"2024-03-30T13:00:58.231919+0000","last_peered":"2024-03-30T13:00:58.231919+0000","last_clean":"2024-03-30T13:00:58.231919+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T13:00:58.231919+0000","last_undegraded":"2024-03-30T13:00:58.231919+0000","last_fullsized":"2024-03-30T13:00:58.231919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:00:53.751391+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T13:00:53.751391+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:04.418173+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":314,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-30T13:00:58.234087+0000","last_change":"2024-03-30T13:00:50.897620+0000","last_active":"2024-03-30T13:00:58.234087+0000","last_peered":"2024-03-30T13:00:58.234087+0000","last_clean":"2024-03-30T13:00:58.234087+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T13:00:58.234087+0000","last_undegraded":"2024-03-30T13:00:58.234087+0000","last_fullsized":"2024-03-30T13:00:58.234087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:00:50.897208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:50.897208+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:00.526987+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":93,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-30T13:00:56.644475+0000","last_change":"2024-03-30T13:00:39.088475+0000","last_active":"2024-03-30T13:00:56.644475+0000","last_peered":"2024-03-30T13:00:56.644475+0000","last_clean":"2024-03-30T13:00:56.644475+0000","last_became_active":"2024-03-30T12:59:55.310542+0000","last_became_peered":"2024-03-30T12:59:55.310542+0000","last_unstale":"2024-03-30T13:00:56.644475+0000","last_undegraded":"2024-03-30T13:00:56.644475+0000","last_fullsized":"2024-03-30T13:00:56.644475+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:00:39.088323+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:00:39.088323+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:05.618294+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":314,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-30T13:00:58.234238+0000","last_change":"2024-03-30T13:00:36.953929+0000","last_active":"2024-03-30T13:00:58.234238+0000","last_peered":"2024-03-30T13:00:58.234238+0000","last_clean":"2024-03-30T13:00:58.234238+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T13:00:58.234238+0000","last_undegraded":"2024-03-30T13:00:58.234238+0000","last_fullsized":"2024-03-30T13:00:58.234238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:00:36.953660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:36.953660+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:48.382597+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":288,"reported_epoch":147,"state":"active+clean","last_fresh":"2024-03-30T13:00:57.228003+0000","last_change":"2024-03-30T13:00:06.700929+0000","last_active":"2024-03-30T13:00:57.228003+0000","last_peered":"2024-03-30T13:00:57.228003+0000","last_clean":"2024-03-30T13:00:57.228003+0000","last_became_active":"2024-03-30T12:58:59.113215+0000","last_became_peered":"2024-03-30T12:58:59.113215+0000","last_unstale":"2024-03-30T13:00:57.228003+0000","last_undegraded":"2024-03-30T13:00:57.228003+0000","last_fullsized":"2024-03-30T13:00:57.228003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:00:06.700643+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:58:34.875943+0000","last_clean_scrub_stamp":"2024-03-30T13:00:06.700643+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:25.056072+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":121445765,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":95059968,"data_stored":95059968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509508,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43856,"kb_used_data":16836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94327984,"statfs":{"total":96636764160,"available":96591855616,"internally_reserved":0,"allocated":17240064,"data_stored":16250445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542212,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47380,"kb_used_data":20360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94324460,"statfs":{"total":96636764160,"available":96588247040,"internally_reserved":0,"allocated":20848640,"data_stored":19859021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":135,"seq":579820584965,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45436,"kb_used_data":13296,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94326404,"statfs":{"total":96636764160,"available":96590237696,"internally_reserved":0,"allocated":13615104,"data_stored":12618734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6047,"internal_metadata":32893025},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509509,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39244,"kb_used_data":8064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332596,"statfs":{"total":96636764160,"available":96596578304,"internally_reserved":0,"allocated":8257536,"data_stored":7261166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285594,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34416,"kb_used_data":6884,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94337424,"statfs":{"total":96636764160,"available":96601522176,"internally_reserved":0,"allocated":7049216,"data_stored":6066348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":28176082},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509509,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42248,"kb_used_data":15216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329592,"statfs":{"total":96636764160,"available":96593502208,"internally_reserved":0,"allocated":15581184,"data_stored":14598316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":107,"seq":459561500688,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43888,"kb_used_data":11748,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94327952,"statfs":{"total":96636764160,"available":96591822848,"internally_reserved":0,"allocated":12029952,"data_stored":11053885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6042,"internal_metadata":32893030},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574917,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41228,"kb_used_data":14204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330612,"statfs":{"total":96636764160,"available":96594546688,"internally_reserved":0,"allocated":14544896,"data_stored":13568829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19079168,"data_stored":19079168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:01:03.202 INFO:tasks.thrashosds.thrasher:Choosing PG 1.a with acting set [5, 2, 6] 2024-03-30T13:01:03.202 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 2, 3, 5, 6, 1, 7, 4] 2024-03-30T13:01:03.203 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-30T13:01:03.203 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:01:03.270 INFO:tasks.ceph.osd.5:Stopped 2024-03-30T13:01:03.270 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 2, 3, 5, 6, 1, 7, 4] 2024-03-30T13:01:03.270 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 5 2024-03-30T13:01:03.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:03.381+0000 7f299c2a9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:03.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:03.397+0000 7f299c2a9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:03.571 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:01:03.569+0000 7fbb38f08640 -1 mon.a@0(leader).osd e152 definitely_dead 0 2024-03-30T13:01:04.276 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.5. 2024-03-30T13:01:04.307 INFO:tasks.thrashosds.thrasher:dead_osds=[5], live_osds=[0, 2, 3, 6, 1, 7, 4] 2024-03-30T13:01:04.307 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:01:04.307 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:01:04.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:04.421+0000 7ff326ced640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:04.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:04.433+0000 7ff326ced640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:04.608 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:01:04.608 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:01:04.627 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":312,"stamp":"2024-03-30T13:01:04.277547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":121445765,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":290492,"kb_used_data":86416,"kb_used_omap":22,"kb_used_meta":203945,"kb_avail":660312388,"statfs":{"total":676457349120,"available":676159885312,"internally_reserved":0,"allocated":88489984,"data_stored":81494841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22868,"internal_metadata":208840364},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.402513"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":328,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-30T13:00:58.231785+0000","last_change":"2024-03-30T13:00:46.110671+0000","last_active":"2024-03-30T13:00:58.231785+0000","last_peered":"2024-03-30T13:00:58.231785+0000","last_clean":"2024-03-30T13:00:58.231785+0000","last_became_active":"2024-03-30T13:00:46.109346+0000","last_became_peered":"2024-03-30T13:00:46.109346+0000","last_unstale":"2024-03-30T13:00:58.231785+0000","last_undegraded":"2024-03-30T13:00:58.231785+0000","last_fullsized":"2024-03-30T13:00:58.231785+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T12:59:56.672842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:56.672842+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:08.297620+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":291,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-30T13:00:59.247512+0000","last_change":"2024-03-30T13:00:58.359297+0000","last_active":"2024-03-30T13:00:59.247512+0000","last_peered":"2024-03-30T13:00:59.247512+0000","last_clean":"2024-03-30T13:00:59.247512+0000","last_became_active":"2024-03-30T13:00:46.110558+0000","last_became_peered":"2024-03-30T13:00:46.110558+0000","last_unstale":"2024-03-30T13:00:59.247512+0000","last_undegraded":"2024-03-30T13:00:59.247512+0000","last_fullsized":"2024-03-30T13:00:59.247512+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:00:58.359040+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:00:58.359040+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:20.772622+0000","scrub_duration":927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":128,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-30T13:01:03.277689+0000","last_change":"2024-03-30T13:01:03.022521+0000","last_active":"2024-03-30T13:01:03.277689+0000","last_peered":"2024-03-30T13:01:03.277689+0000","last_clean":"2024-03-30T13:01:03.277689+0000","last_became_active":"2024-03-30T13:00:46.112427+0000","last_became_peered":"2024-03-30T13:00:46.112427+0000","last_unstale":"2024-03-30T13:01:03.277689+0000","last_undegraded":"2024-03-30T13:01:03.277689+0000","last_fullsized":"2024-03-30T13:01:03.277689+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:01:03.022279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:03.022279+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:08.969141+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":106,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-30T13:01:03.277582+0000","last_change":"2024-03-30T13:00:35.159895+0000","last_active":"2024-03-30T13:01:03.277582+0000","last_peered":"2024-03-30T13:01:03.277582+0000","last_clean":"2024-03-30T13:01:03.277582+0000","last_became_active":"2024-03-30T12:59:55.316713+0000","last_became_peered":"2024-03-30T12:59:55.316713+0000","last_unstale":"2024-03-30T13:01:03.277582+0000","last_undegraded":"2024-03-30T13:01:03.277582+0000","last_fullsized":"2024-03-30T13:01:03.277582+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:00:35.159640+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:00:35.159640+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:04.854390+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":294,"reported_epoch":148,"state":"stale+active+clean","last_fresh":"2024-03-30T13:00:58.234115+0000","last_change":"2024-03-30T13:00:01.952302+0000","last_active":"2024-03-30T13:00:58.234115+0000","last_peered":"2024-03-30T13:00:58.234115+0000","last_clean":"2024-03-30T13:00:58.234115+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T13:01:03.275113+0000","last_undegraded":"2024-03-30T13:00:58.234115+0000","last_fullsized":"2024-03-30T13:00:58.234115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:00:01.951942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:01.951942+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:08.852178+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":320,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-30T13:01:02.425312+0000","last_change":"2024-03-30T13:00:46.414441+0000","last_active":"2024-03-30T13:01:02.425312+0000","last_peered":"2024-03-30T13:01:02.425312+0000","last_clean":"2024-03-30T13:01:02.425312+0000","last_became_active":"2024-03-30T12:55:43.927249+0000","last_became_peered":"2024-03-30T12:55:43.927249+0000","last_unstale":"2024-03-30T13:01:02.425312+0000","last_undegraded":"2024-03-30T13:01:02.425312+0000","last_fullsized":"2024-03-30T13:01:02.425312+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:00:46.414102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:46.414102+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:05.949447+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":147,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-30T13:01:03.277586+0000","last_change":"2024-03-30T13:00:46.115096+0000","last_active":"2024-03-30T13:01:03.277586+0000","last_peered":"2024-03-30T13:01:03.277586+0000","last_clean":"2024-03-30T13:01:03.277586+0000","last_became_active":"2024-03-30T13:00:46.114131+0000","last_became_peered":"2024-03-30T13:00:46.114131+0000","last_unstale":"2024-03-30T13:01:03.277586+0000","last_undegraded":"2024-03-30T13:01:03.277586+0000","last_fullsized":"2024-03-30T13:01:03.277586+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:00:34.155844+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:47.134436+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":307,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-30T13:01:01.268339+0000","last_change":"2024-03-30T12:59:59.675214+0000","last_active":"2024-03-30T13:01:01.268339+0000","last_peered":"2024-03-30T13:01:01.268339+0000","last_clean":"2024-03-30T13:01:01.268339+0000","last_became_active":"2024-03-30T12:58:59.114041+0000","last_became_peered":"2024-03-30T12:58:59.114041+0000","last_unstale":"2024-03-30T13:01:01.268339+0000","last_undegraded":"2024-03-30T13:01:01.268339+0000","last_fullsized":"2024-03-30T13:01:01.268339+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T12:59:59.675050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:59.675050+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:20.917814+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7344560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":361,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-30T13:01:02.425429+0000","last_change":"2024-03-30T13:00:49.392191+0000","last_active":"2024-03-30T13:01:02.425429+0000","last_peered":"2024-03-30T13:01:02.425429+0000","last_clean":"2024-03-30T13:01:02.425429+0000","last_became_active":"2024-03-30T12:59:55.305411+0000","last_became_peered":"2024-03-30T12:59:55.305411+0000","last_unstale":"2024-03-30T13:01:02.425429+0000","last_undegraded":"2024-03-30T13:01:02.425429+0000","last_fullsized":"2024-03-30T13:01:02.425429+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:00:49.391955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:49.391955+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:04.091733+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":20302592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":219,"reported_epoch":149,"state":"active+clean","last_fresh":"2024-03-30T13:00:59.247404+0000","last_change":"2024-03-30T13:00:46.110349+0000","last_active":"2024-03-30T13:00:59.247404+0000","last_peered":"2024-03-30T13:00:59.247404+0000","last_clean":"2024-03-30T13:00:59.247404+0000","last_became_active":"2024-03-30T13:00:46.109054+0000","last_became_peered":"2024-03-30T13:00:46.109054+0000","last_unstale":"2024-03-30T13:00:59.247404+0000","last_undegraded":"2024-03-30T13:00:59.247404+0000","last_fullsized":"2024-03-30T13:00:59.247404+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T12:59:42.816452+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:12.073828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":355,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-30T13:00:58.231773+0000","last_change":"2024-03-30T13:00:46.107422+0000","last_active":"2024-03-30T13:00:58.231773+0000","last_peered":"2024-03-30T13:00:58.231773+0000","last_clean":"2024-03-30T13:00:58.231773+0000","last_became_active":"2024-03-30T13:00:46.106211+0000","last_became_peered":"2024-03-30T13:00:46.106211+0000","last_unstale":"2024-03-30T13:00:58.231773+0000","last_undegraded":"2024-03-30T13:00:58.231773+0000","last_fullsized":"2024-03-30T13:00:58.231773+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T12:59:37.603454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T12:59:37.603454+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:03.418078+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25479082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":324,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-30T13:00:58.231919+0000","last_change":"2024-03-30T13:00:53.751657+0000","last_active":"2024-03-30T13:00:58.231919+0000","last_peered":"2024-03-30T13:00:58.231919+0000","last_clean":"2024-03-30T13:00:58.231919+0000","last_became_active":"2024-03-30T12:55:43.930712+0000","last_became_peered":"2024-03-30T12:55:43.930712+0000","last_unstale":"2024-03-30T13:00:58.231919+0000","last_undegraded":"2024-03-30T13:00:58.231919+0000","last_fullsized":"2024-03-30T13:00:58.231919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:00:53.751391+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T13:00:53.751391+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:04.418173+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":314,"reported_epoch":148,"state":"stale+active+clean","last_fresh":"2024-03-30T13:00:58.234087+0000","last_change":"2024-03-30T13:00:50.897620+0000","last_active":"2024-03-30T13:00:58.234087+0000","last_peered":"2024-03-30T13:00:58.234087+0000","last_clean":"2024-03-30T13:00:58.234087+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T13:01:03.275118+0000","last_undegraded":"2024-03-30T13:00:58.234087+0000","last_fullsized":"2024-03-30T13:00:58.234087+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:00:50.897208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:50.897208+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:00.526987+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":104,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-30T13:01:03.277586+0000","last_change":"2024-03-30T13:00:39.088475+0000","last_active":"2024-03-30T13:01:03.277586+0000","last_peered":"2024-03-30T13:01:03.277586+0000","last_clean":"2024-03-30T13:01:03.277586+0000","last_became_active":"2024-03-30T12:59:55.310542+0000","last_became_peered":"2024-03-30T12:59:55.310542+0000","last_unstale":"2024-03-30T13:01:03.277586+0000","last_undegraded":"2024-03-30T13:01:03.277586+0000","last_fullsized":"2024-03-30T13:01:03.277586+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:00:39.088323+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:00:39.088323+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:05.618294+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":314,"reported_epoch":148,"state":"stale+active+clean","last_fresh":"2024-03-30T13:00:58.234238+0000","last_change":"2024-03-30T13:00:36.953929+0000","last_active":"2024-03-30T13:00:58.234238+0000","last_peered":"2024-03-30T13:00:58.234238+0000","last_clean":"2024-03-30T13:00:58.234238+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T13:01:03.275123+0000","last_undegraded":"2024-03-30T13:00:58.234238+0000","last_fullsized":"2024-03-30T13:00:58.234238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:00:36.953660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:36.953660+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:48.382597+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":296,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-03-30T13:01:01.268275+0000","last_change":"2024-03-30T13:00:06.700929+0000","last_active":"2024-03-30T13:01:01.268275+0000","last_peered":"2024-03-30T13:01:01.268275+0000","last_clean":"2024-03-30T13:01:01.268275+0000","last_became_active":"2024-03-30T12:58:59.113215+0000","last_became_peered":"2024-03-30T12:58:59.113215+0000","last_unstale":"2024-03-30T13:01:01.268275+0000","last_undegraded":"2024-03-30T13:01:01.268275+0000","last_fullsized":"2024-03-30T13:01:01.268275+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:00:06.700643+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:58:34.875943+0000","last_clean_scrub_stamp":"2024-03-30T13:00:06.700643+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:25.056072+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":95,"num_read_kb":4616,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":121445765,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":95059968,"data_stored":95059968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509508,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43856,"kb_used_data":16836,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94327984,"statfs":{"total":96636764160,"available":96591855616,"internally_reserved":0,"allocated":17240064,"data_stored":16250445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":135,"seq":579820584965,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45436,"kb_used_data":13296,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94326404,"statfs":{"total":96636764160,"available":96590237696,"internally_reserved":0,"allocated":13615104,"data_stored":12618734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6047,"internal_metadata":32893025},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509509,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39244,"kb_used_data":8064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332596,"statfs":{"total":96636764160,"available":96596578304,"internally_reserved":0,"allocated":8257536,"data_stored":7261166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285595,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34468,"kb_used_data":6932,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94337372,"statfs":{"total":96636764160,"available":96601468928,"internally_reserved":0,"allocated":7098368,"data_stored":6088396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":28176082},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509509,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42248,"kb_used_data":15216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329592,"statfs":{"total":96636764160,"available":96593502208,"internally_reserved":0,"allocated":15581184,"data_stored":14598316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":107,"seq":459561500689,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43956,"kb_used_data":11808,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94327884,"statfs":{"total":96636764160,"available":96591753216,"internally_reserved":0,"allocated":12091392,"data_stored":11081420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6042,"internal_metadata":32893030},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574918,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41284,"kb_used_data":14264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330556,"statfs":{"total":96636764160,"available":96594489344,"internally_reserved":0,"allocated":14606336,"data_stored":13596364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19079168,"data_stored":19079168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:01:04.628 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:01:29.629 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:01:29.629 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:01:29.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:29.753+0000 7f75ab0a4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:29.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:29.765+0000 7f75ab0a4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:29.947 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:01:29.947 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:01:29.965 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-03-30T13:01:29.884798+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":194790490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":319016,"kb_used_data":106652,"kb_used_omap":22,"kb_used_meta":212265,"kb_avail":660283864,"statfs":{"total":676457349120,"available":676130676736,"internally_reserved":0,"allocated":109211648,"data_stored":102001438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22868,"internal_metadata":217360044},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003887"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":359,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-30T13:01:22.929002+0000","last_change":"2024-03-30T13:01:22.929079+0000","last_active":"2024-03-30T13:01:22.929002+0000","last_peered":"2024-03-30T13:01:22.929002+0000","last_clean":"2024-03-30T13:01:22.929002+0000","last_became_active":"2024-03-30T13:01:05.298955+0000","last_became_peered":"2024-03-30T13:01:05.298955+0000","last_unstale":"2024-03-30T13:01:22.929002+0000","last_undegraded":"2024-03-30T13:01:22.929002+0000","last_fullsized":"2024-03-30T13:01:22.929002+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:01:22.928753+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:22.928753+0000","last_clean_scrub_stamp":"2024-03-30T13:01:22.928753+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:38.706818+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":13764735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":301,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-30T13:01:05.286949+0000","last_change":"2024-03-30T13:00:58.359297+0000","last_active":"2024-03-30T13:01:05.286949+0000","last_peered":"2024-03-30T13:01:05.286949+0000","last_clean":"2024-03-30T13:01:05.286949+0000","last_became_active":"2024-03-30T13:00:46.110558+0000","last_became_peered":"2024-03-30T13:00:46.110558+0000","last_unstale":"2024-03-30T13:01:05.286949+0000","last_undegraded":"2024-03-30T13:01:05.286949+0000","last_fullsized":"2024-03-30T13:01:05.286949+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:00:58.359040+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:00:58.359040+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:20.772622+0000","scrub_duration":927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":132,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-30T13:01:05.291341+0000","last_change":"2024-03-30T13:01:03.022521+0000","last_active":"2024-03-30T13:01:05.291341+0000","last_peered":"2024-03-30T13:01:05.291341+0000","last_clean":"2024-03-30T13:01:05.291341+0000","last_became_active":"2024-03-30T13:00:46.112427+0000","last_became_peered":"2024-03-30T13:00:46.112427+0000","last_unstale":"2024-03-30T13:01:05.291341+0000","last_undegraded":"2024-03-30T13:01:05.291341+0000","last_fullsized":"2024-03-30T13:01:05.291341+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:01:03.022279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:03.022279+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:08.969141+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":110,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-30T13:01:05.290998+0000","last_change":"2024-03-30T13:00:35.159895+0000","last_active":"2024-03-30T13:01:05.290998+0000","last_peered":"2024-03-30T13:01:05.290998+0000","last_clean":"2024-03-30T13:01:05.290998+0000","last_became_active":"2024-03-30T12:59:55.316713+0000","last_became_peered":"2024-03-30T12:59:55.316713+0000","last_unstale":"2024-03-30T13:01:05.290998+0000","last_undegraded":"2024-03-30T13:01:05.290998+0000","last_fullsized":"2024-03-30T13:01:05.290998+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:00:35.159640+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:00:35.159640+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:04.854390+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":23,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-30T13:01:07.413362+0000","last_change":"2024-03-30T13:01:07.413433+0000","last_active":"2024-03-30T13:01:07.413362+0000","last_peered":"2024-03-30T13:01:07.413362+0000","last_clean":"2024-03-30T13:01:07.413362+0000","last_became_active":"2024-03-30T13:01:05.296945+0000","last_became_peered":"2024-03-30T13:01:05.296945+0000","last_unstale":"2024-03-30T13:01:07.413362+0000","last_undegraded":"2024-03-30T13:01:07.413362+0000","last_fullsized":"2024-03-30T13:01:07.413362+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:01:07.413186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:07.413186+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:27.458032+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":343,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-30T13:01:05.988028+0000","last_change":"2024-03-30T13:01:05.988028+0000","last_active":"2024-03-30T13:01:05.988028+0000","last_peered":"2024-03-30T13:01:05.988028+0000","last_clean":"2024-03-30T13:01:05.988028+0000","last_became_active":"2024-03-30T13:01:05.303554+0000","last_became_peered":"2024-03-30T13:01:05.303554+0000","last_unstale":"2024-03-30T13:01:05.988028+0000","last_undegraded":"2024-03-30T13:01:05.988028+0000","last_fullsized":"2024-03-30T13:01:05.988028+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:00:46.414102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:46.414102+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:01.515314+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11394981,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":151,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-30T13:01:05.290903+0000","last_change":"2024-03-30T13:00:46.115096+0000","last_active":"2024-03-30T13:01:05.290903+0000","last_peered":"2024-03-30T13:01:05.290903+0000","last_clean":"2024-03-30T13:01:05.290903+0000","last_became_active":"2024-03-30T13:00:46.114131+0000","last_became_peered":"2024-03-30T13:00:46.114131+0000","last_unstale":"2024-03-30T13:01:05.290903+0000","last_undegraded":"2024-03-30T13:01:05.290903+0000","last_fullsized":"2024-03-30T13:01:05.290903+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:00:34.155844+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:47.134436+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":334,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-30T13:01:16.584513+0000","last_change":"2024-03-30T13:01:16.584590+0000","last_active":"2024-03-30T13:01:16.584513+0000","last_peered":"2024-03-30T13:01:16.584513+0000","last_clean":"2024-03-30T13:01:16.584513+0000","last_became_active":"2024-03-30T13:01:05.296737+0000","last_became_peered":"2024-03-30T13:01:05.296737+0000","last_unstale":"2024-03-30T13:01:16.584513+0000","last_undegraded":"2024-03-30T13:01:16.584513+0000","last_fullsized":"2024-03-30T13:01:16.584513+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:01:16.584341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:16.584341+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:26.129089+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14689120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":378,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-30T13:01:06.127860+0000","last_change":"2024-03-30T13:01:06.127860+0000","last_active":"2024-03-30T13:01:06.127860+0000","last_peered":"2024-03-30T13:01:06.127860+0000","last_clean":"2024-03-30T13:01:06.127860+0000","last_became_active":"2024-03-30T13:01:05.296260+0000","last_became_peered":"2024-03-30T13:01:05.296260+0000","last_unstale":"2024-03-30T13:01:06.127860+0000","last_undegraded":"2024-03-30T13:01:06.127860+0000","last_fullsized":"2024-03-30T13:01:06.127860+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:00:49.391955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:49.391955+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:13.908960+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25378240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":237,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-30T13:01:13.453295+0000","last_change":"2024-03-30T13:01:13.453369+0000","last_active":"2024-03-30T13:01:13.453295+0000","last_peered":"2024-03-30T13:01:13.453295+0000","last_clean":"2024-03-30T13:01:13.453295+0000","last_became_active":"2024-03-30T13:00:46.109054+0000","last_became_peered":"2024-03-30T13:00:46.109054+0000","last_unstale":"2024-03-30T13:01:13.453295+0000","last_undegraded":"2024-03-30T13:01:13.453295+0000","last_fullsized":"2024-03-30T13:01:13.453295+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:01:13.453092+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:01:13.453092+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:24.209616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":375,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-30T13:01:05.282717+0000","last_change":"2024-03-30T13:01:03.880400+0000","last_active":"2024-03-30T13:01:05.282717+0000","last_peered":"2024-03-30T13:01:05.282717+0000","last_clean":"2024-03-30T13:01:05.282717+0000","last_became_active":"2024-03-30T13:00:46.106211+0000","last_became_peered":"2024-03-30T13:00:46.106211+0000","last_unstale":"2024-03-30T13:01:05.282717+0000","last_undegraded":"2024-03-30T13:01:05.282717+0000","last_fullsized":"2024-03-30T13:01:05.282717+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:01:03.880142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:03.880142+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:16.907695+0000","scrub_duration":251,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25479082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":359,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-30T13:01:05.499778+0000","last_change":"2024-03-30T13:01:05.499778+0000","last_active":"2024-03-30T13:01:05.499778+0000","last_peered":"2024-03-30T13:01:05.499778+0000","last_clean":"2024-03-30T13:01:05.499778+0000","last_became_active":"2024-03-30T13:01:05.296400+0000","last_became_peered":"2024-03-30T13:01:05.296400+0000","last_unstale":"2024-03-30T13:01:05.499778+0000","last_undegraded":"2024-03-30T13:01:05.499778+0000","last_fullsized":"2024-03-30T13:01:05.499778+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:00:53.751391+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T13:00:53.751391+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:57.658617+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":21,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-30T13:01:05.854260+0000","last_change":"2024-03-30T13:01:05.854260+0000","last_active":"2024-03-30T13:01:05.854260+0000","last_peered":"2024-03-30T13:01:05.854260+0000","last_clean":"2024-03-30T13:01:05.854260+0000","last_became_active":"2024-03-30T13:01:05.297245+0000","last_became_peered":"2024-03-30T13:01:05.297245+0000","last_unstale":"2024-03-30T13:01:05.854260+0000","last_undegraded":"2024-03-30T13:01:05.854260+0000","last_fullsized":"2024-03-30T13:01:05.854260+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:00:50.897208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:50.897208+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:51.957560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":108,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-30T13:01:05.290878+0000","last_change":"2024-03-30T13:00:39.088475+0000","last_active":"2024-03-30T13:01:05.290878+0000","last_peered":"2024-03-30T13:01:05.290878+0000","last_clean":"2024-03-30T13:01:05.290878+0000","last_became_active":"2024-03-30T12:59:55.310542+0000","last_became_peered":"2024-03-30T12:59:55.310542+0000","last_unstale":"2024-03-30T13:01:05.290878+0000","last_undegraded":"2024-03-30T13:01:05.290878+0000","last_fullsized":"2024-03-30T13:01:05.290878+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:00:39.088323+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:00:39.088323+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:05.618294+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":21,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-30T13:01:06.300143+0000","last_change":"2024-03-30T13:01:06.300143+0000","last_active":"2024-03-30T13:01:06.300143+0000","last_peered":"2024-03-30T13:01:06.300143+0000","last_clean":"2024-03-30T13:01:06.300143+0000","last_became_active":"2024-03-30T13:01:05.299498+0000","last_became_peered":"2024-03-30T13:01:05.299498+0000","last_unstale":"2024-03-30T13:01:06.300143+0000","last_undegraded":"2024-03-30T13:01:06.300143+0000","last_fullsized":"2024-03-30T13:01:06.300143+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:00:36.953660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:36.953660+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:01:43.353957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":322,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-30T13:01:27.958575+0000","last_change":"2024-03-30T13:01:27.958648+0000","last_active":"2024-03-30T13:01:27.958575+0000","last_peered":"2024-03-30T13:01:27.958575+0000","last_clean":"2024-03-30T13:01:27.958575+0000","last_became_active":"2024-03-30T13:01:05.299075+0000","last_became_peered":"2024-03-30T13:01:05.299075+0000","last_unstale":"2024-03-30T13:01:27.958575+0000","last_undegraded":"2024-03-30T13:01:27.958575+0000","last_fullsized":"2024-03-30T13:01:27.958575+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:01:27.958371+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:31.516127+0000","scrub_duration":451,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5226911,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":194790490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":115388416,"data_stored":115388416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509514,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53648,"kb_used_data":22472,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94318192,"statfs":{"total":96636764160,"available":96581828608,"internally_reserved":0,"allocated":23011328,"data_stored":21981298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":135,"seq":579820584970,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49988,"kb_used_data":17852,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94321852,"statfs":{"total":96636764160,"available":96585576448,"internally_reserved":0,"allocated":18280448,"data_stored":17250418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6047,"internal_metadata":32893025},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509515,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41928,"kb_used_data":10752,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329912,"statfs":{"total":96636764160,"available":96593829888,"internally_reserved":0,"allocated":11010048,"data_stored":9980018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285600,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43868,"kb_used_data":12180,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94327972,"statfs":{"total":96636764160,"available":96591843328,"internally_reserved":0,"allocated":12472320,"data_stored":11442290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509515,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42316,"kb_used_data":15300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329524,"statfs":{"total":96636764160,"available":96593432576,"internally_reserved":0,"allocated":15667200,"data_stored":14637170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":107,"seq":459561500694,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43992,"kb_used_data":11844,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94327848,"statfs":{"total":96636764160,"available":96591716352,"internally_reserved":0,"allocated":12128256,"data_stored":11098226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6042,"internal_metadata":32893030},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574923,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43276,"kb_used_data":16252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94328564,"statfs":{"total":96636764160,"available":96592449536,"internally_reserved":0,"allocated":16642048,"data_stored":15612018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10629120,"data_stored":10629120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9166848,"data_stored":9166848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":14798848,"data_stored":14798848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16437248,"data_stored":16437248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19079168,"data_stored":19079168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":21168128,"data_stored":21168128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:01:29.966 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:01:29.966 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:01:29.967 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-03-30T13:01:29.967 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-30T13:01:29.967 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-30T13:01:29.969 INFO:tasks.ceph.osd.5:Started 2024-03-30T13:01:29.970 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-30T13:01:30.038 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:01:30.031+0000 7fa0dd9137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:30.046 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:01:30.039+0000 7fa0dd9137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:30.046 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:01:30.039+0000 7fa0dd9137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:30.091 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T13:01:30.091 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T13:01:30.091 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T13:01:30.091 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T13:01:30.091 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T13:01:30.092 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T13:01:30.092 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-03-30T13:01:30.092 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-03-30T13:01:30.092 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-03-30T13:01:30.092 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-03-30T13:01:30.092 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-03-30T13:01:30.092 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T13:01:30.093 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:01:30.093 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-03-30T13:01:30.580 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:01:30.575+0000 7fa0dd9137c0 -1 Falling back to public interface 2024-03-30T13:01:34.346 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:01:34.339+0000 7fa0dd9137c0 -1 osd.5 151 log_to_monitors true 2024-03-30T13:01:35.095 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-30T13:01:35.214 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T13:01:35.214 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T13:01:35.214 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T13:01:35.214 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T13:01:35.214 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T13:01:35.214 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T13:01:35.215 INFO:teuthology.orchestra.run.smithi082.stderr:bluefs debug_inject_read_zeros 2024-03-30T13:01:35.215 INFO:teuthology.orchestra.run.smithi082.stderr:bluefs files list 2024-03-30T13:01:35.215 INFO:teuthology.orchestra.run.smithi082.stderr:bluefs stats 2024-03-30T13:01:35.215 INFO:teuthology.orchestra.run.smithi082.stderr:bluestore allocator dump block 2024-03-30T13:01:35.215 INFO:teuthology.orchestra.run.smithi082.stderr:bluestore allocator fragmentation block 2024-03-30T13:01:35.215 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T13:01:35.216 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:01:35.216 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-03-30T13:01:36.343 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:01:36.339+0000 7fa0d16e0640 -1 osd.5 151 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:01:40.218 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-30T13:01:40.349 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T13:01:40.349 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T13:01:40.349 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T13:01:40.349 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T13:01:40.363 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:01:40.363 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-03-30T13:01:40.363 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 5 2024-03-30T13:01:40.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:40.488+0000 7f5f0cc94640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:40.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:40.504+0000 7f5f0cc94640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:40.687 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:01:40.684+0000 7fbb38f08640 -1 mon.a@0(leader).osd e156 definitely_dead 0 2024-03-30T13:01:40.959 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.5. 2024-03-30T13:01:40.987 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-03-30T13:01:40.987 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:01:41.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:41.096+0000 7f8aa5209640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:41.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:41.112+0000 7f8aa5209640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:41.972 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:01:42.000 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:01:42.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:42.124+0000 7f01ddbbf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:42.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:42.140+0000 7f01ddbbf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:42.971 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:01:42.995 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:01:43.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:43.112+0000 7fc064b08640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:43.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:43.128+0000 7fc064b08640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:43.983 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:01:44.010 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:01:44.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:44.128+0000 7f3ff1f1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:44.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:44.144+0000 7f3ff1f1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:44.999 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:01:45.021 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:01:45.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:45.136+0000 7fc82fb82640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:45.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:45.152+0000 7fc82fb82640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:01:46.020 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:01:46.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:46.132+0000 7fbd01e70640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:46.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:46.144+0000 7fbd01e70640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:47.011 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:01:47.039 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:01:47.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:47.156+0000 7fa719c3a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:47.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:47.172+0000 7fa719c3a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:48.015 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:01:48.046 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:01:48.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:48.164+0000 7fbb6f415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:48.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:48.384+0000 7fbb6c9b0640 0 --2- 172.21.15.67:0/3483159311 >> v2:172.21.15.82:3300/0 conn(0x7fbb68143050 0x7fbb680a9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-03-30T13:01:48.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:48.388+0000 7fbb6f415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:49.043 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:01:49.071 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:01:49.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:49.188+0000 7f556506b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:49.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:49.204+0000 7f556506b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:50.049 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:01:50.075 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:01:50.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:50.188+0000 7fbe8a19c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:50.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:50.204+0000 7fbe8a19c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:51.067 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:01:51.091 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:01:51.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:51.204+0000 7f622b9db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:51.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:51.220+0000 7f622b9db640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:52.079 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:01:52.106 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:01:52.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:52.224+0000 7fc607e0a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:52.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:52.240+0000 7fc607e0a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:53.084 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:01:53.113 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:01:53.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:53.224+0000 7f436e3f3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:53.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:53.240+0000 7f436e3f3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:54.117 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:01:54.143 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:01:54.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:54.260+0000 7fb9da8da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:54.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:54.276+0000 7fb9da8da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:55.116 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:01:55.142 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:01:55.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:55.260+0000 7f4bb6f93640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:55.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:55.276+0000 7f4bb6f93640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:56.132 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:01:56.160 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:01:56.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:56.276+0000 7f91989b9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:56.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:56.292+0000 7f91989b9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:57.134 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:01:57.165 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:01:57.165 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:01:57.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:57.284+0000 7fe07eb0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:57.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:57.296+0000 7fe07eb0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:57.471 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:01:57.471 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:01:57.489 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2024-03-30T13:01:57.131833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":247664586,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":358096,"kb_used_data":109844,"kb_used_omap":26,"kb_used_meta":248101,"kb_avail":754616624,"statfs":{"total":773094113280,"available":772727422976,"internally_reserved":0,"allocated":112480256,"data_stored":103466476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27273,"internal_metadata":254055799},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.042305"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":401,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-30T13:01:53.149419+0000","last_change":"2024-03-30T13:01:42.155603+0000","last_active":"2024-03-30T13:01:53.149419+0000","last_peered":"2024-03-30T13:01:53.149419+0000","last_clean":"2024-03-30T13:01:53.149419+0000","last_became_active":"2024-03-30T13:01:41.987359+0000","last_became_peered":"2024-03-30T13:01:41.987359+0000","last_unstale":"2024-03-30T13:01:53.149419+0000","last_undegraded":"2024-03-30T13:01:53.149419+0000","last_fullsized":"2024-03-30T13:01:53.149419+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:01:22.928753+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:22.928753+0000","last_clean_scrub_stamp":"2024-03-30T13:01:22.928753+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:51.214009+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18352980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":332,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-30T13:01:54.674496+0000","last_change":"2024-03-30T13:00:58.359297+0000","last_active":"2024-03-30T13:01:54.674496+0000","last_peered":"2024-03-30T13:01:54.674496+0000","last_clean":"2024-03-30T13:01:54.674496+0000","last_became_active":"2024-03-30T13:00:46.110558+0000","last_became_peered":"2024-03-30T13:00:46.110558+0000","last_unstale":"2024-03-30T13:01:54.674496+0000","last_undegraded":"2024-03-30T13:01:54.674496+0000","last_fullsized":"2024-03-30T13:01:54.674496+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:00:58.359040+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:00:58.359040+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:20.772622+0000","scrub_duration":927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":160,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-30T13:01:52.941864+0000","last_change":"2024-03-30T13:01:03.022521+0000","last_active":"2024-03-30T13:01:52.941864+0000","last_peered":"2024-03-30T13:01:52.941864+0000","last_clean":"2024-03-30T13:01:52.941864+0000","last_became_active":"2024-03-30T13:00:46.112427+0000","last_became_peered":"2024-03-30T13:00:46.112427+0000","last_unstale":"2024-03-30T13:01:52.941864+0000","last_undegraded":"2024-03-30T13:01:52.941864+0000","last_fullsized":"2024-03-30T13:01:52.941864+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:01:03.022279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:03.022279+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:08.969141+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":138,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-30T13:01:52.941490+0000","last_change":"2024-03-30T13:00:35.159895+0000","last_active":"2024-03-30T13:01:52.941490+0000","last_peered":"2024-03-30T13:01:52.941490+0000","last_clean":"2024-03-30T13:01:52.941490+0000","last_became_active":"2024-03-30T12:59:55.316713+0000","last_became_peered":"2024-03-30T12:59:55.316713+0000","last_unstale":"2024-03-30T13:01:52.941490+0000","last_undegraded":"2024-03-30T13:01:52.941490+0000","last_fullsized":"2024-03-30T13:01:52.941490+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:00:35.159640+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:00:35.159640+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:04.854390+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":40,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-30T13:01:55.553779+0000","last_change":"2024-03-30T13:01:42.334934+0000","last_active":"2024-03-30T13:01:55.553779+0000","last_peered":"2024-03-30T13:01:55.553779+0000","last_clean":"2024-03-30T13:01:55.553779+0000","last_became_active":"2024-03-30T13:01:41.984313+0000","last_became_peered":"2024-03-30T13:01:41.984313+0000","last_unstale":"2024-03-30T13:01:55.553779+0000","last_undegraded":"2024-03-30T13:01:55.553779+0000","last_fullsized":"2024-03-30T13:01:55.553779+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:01:07.413186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:07.413186+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:35.302199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":387,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-03-30T13:01:51.755169+0000","last_change":"2024-03-30T13:01:43.093050+0000","last_active":"2024-03-30T13:01:51.755169+0000","last_peered":"2024-03-30T13:01:51.755169+0000","last_clean":"2024-03-30T13:01:51.755169+0000","last_became_active":"2024-03-30T13:01:41.987920+0000","last_became_peered":"2024-03-30T13:01:41.987920+0000","last_unstale":"2024-03-30T13:01:51.755169+0000","last_undegraded":"2024-03-30T13:01:51.755169+0000","last_fullsized":"2024-03-30T13:01:51.755169+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:00:46.414102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:46.414102+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:09.217769+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22789962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":187,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-30T13:01:52.941631+0000","last_change":"2024-03-30T13:01:48.791786+0000","last_active":"2024-03-30T13:01:52.941631+0000","last_peered":"2024-03-30T13:01:52.941631+0000","last_clean":"2024-03-30T13:01:52.941631+0000","last_became_active":"2024-03-30T13:00:46.114131+0000","last_became_peered":"2024-03-30T13:00:46.114131+0000","last_unstale":"2024-03-30T13:01:52.941631+0000","last_undegraded":"2024-03-30T13:01:52.941631+0000","last_fullsized":"2024-03-30T13:01:52.941631+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:01:48.791509+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:01:48.791509+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:04.577057+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":376,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-30T13:01:52.076574+0000","last_change":"2024-03-30T13:01:42.050953+0000","last_active":"2024-03-30T13:01:52.076574+0000","last_peered":"2024-03-30T13:01:52.076574+0000","last_clean":"2024-03-30T13:01:52.076574+0000","last_became_active":"2024-03-30T13:01:41.988151+0000","last_became_peered":"2024-03-30T13:01:41.988151+0000","last_unstale":"2024-03-30T13:01:52.076574+0000","last_undegraded":"2024-03-30T13:01:52.076574+0000","last_fullsized":"2024-03-30T13:01:52.076574+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:01:16.584341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:16.584341+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:35.937908+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":22033680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":416,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-03-30T13:01:51.755411+0000","last_change":"2024-03-30T13:01:42.116405+0000","last_active":"2024-03-30T13:01:51.755411+0000","last_peered":"2024-03-30T13:01:51.755411+0000","last_clean":"2024-03-30T13:01:51.755411+0000","last_became_active":"2024-03-30T13:01:41.989387+0000","last_became_peered":"2024-03-30T13:01:41.989387+0000","last_unstale":"2024-03-30T13:01:51.755411+0000","last_undegraded":"2024-03-30T13:01:51.755411+0000","last_fullsized":"2024-03-30T13:01:51.755411+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:00:49.391955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:49.391955+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:13.856259+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":30453888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":268,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-30T13:01:54.674344+0000","last_change":"2024-03-30T13:01:13.453369+0000","last_active":"2024-03-30T13:01:54.674344+0000","last_peered":"2024-03-30T13:01:54.674344+0000","last_clean":"2024-03-30T13:01:54.674344+0000","last_became_active":"2024-03-30T13:00:46.109054+0000","last_became_peered":"2024-03-30T13:00:46.109054+0000","last_unstale":"2024-03-30T13:01:54.674344+0000","last_undegraded":"2024-03-30T13:01:54.674344+0000","last_fullsized":"2024-03-30T13:01:54.674344+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:01:13.453092+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:01:13.453092+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:24.209616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":405,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-30T13:01:53.149416+0000","last_change":"2024-03-30T13:01:03.880400+0000","last_active":"2024-03-30T13:01:53.149416+0000","last_peered":"2024-03-30T13:01:53.149416+0000","last_clean":"2024-03-30T13:01:53.149416+0000","last_became_active":"2024-03-30T13:00:46.106211+0000","last_became_peered":"2024-03-30T13:00:46.106211+0000","last_unstale":"2024-03-30T13:01:53.149416+0000","last_undegraded":"2024-03-30T13:01:53.149416+0000","last_fullsized":"2024-03-30T13:01:53.149416+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:01:03.880142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:03.880142+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:16.907695+0000","scrub_duration":251,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25479082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":413,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-30T13:01:53.149445+0000","last_change":"2024-03-30T13:01:43.224020+0000","last_active":"2024-03-30T13:01:53.149445+0000","last_peered":"2024-03-30T13:01:53.149445+0000","last_clean":"2024-03-30T13:01:53.149445+0000","last_became_active":"2024-03-30T13:01:41.991185+0000","last_became_peered":"2024-03-30T13:01:41.991185+0000","last_unstale":"2024-03-30T13:01:53.149445+0000","last_undegraded":"2024-03-30T13:01:53.149445+0000","last_fullsized":"2024-03-30T13:01:53.149445+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:00:53.751391+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T13:00:53.751391+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:18.943412+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":54,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-30T13:01:55.553792+0000","last_change":"2024-03-30T13:01:53.533883+0000","last_active":"2024-03-30T13:01:55.553792+0000","last_peered":"2024-03-30T13:01:55.553792+0000","last_clean":"2024-03-30T13:01:55.553792+0000","last_became_active":"2024-03-30T13:01:41.982566+0000","last_became_peered":"2024-03-30T13:01:41.982566+0000","last_unstale":"2024-03-30T13:01:55.553792+0000","last_undegraded":"2024-03-30T13:01:55.553792+0000","last_fullsized":"2024-03-30T13:01:55.553792+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:01:53.533647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:53.533647+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:00.040107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":136,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-30T13:01:52.941492+0000","last_change":"2024-03-30T13:00:39.088475+0000","last_active":"2024-03-30T13:01:52.941492+0000","last_peered":"2024-03-30T13:01:52.941492+0000","last_clean":"2024-03-30T13:01:52.941492+0000","last_became_active":"2024-03-30T12:59:55.310542+0000","last_became_peered":"2024-03-30T12:59:55.310542+0000","last_unstale":"2024-03-30T13:01:52.941492+0000","last_undegraded":"2024-03-30T13:01:52.941492+0000","last_fullsized":"2024-03-30T13:01:52.941492+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:00:39.088323+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:00:39.088323+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:05.618294+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":54,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-30T13:01:55.553869+0000","last_change":"2024-03-30T13:01:51.537905+0000","last_active":"2024-03-30T13:01:55.553869+0000","last_peered":"2024-03-30T13:01:55.553869+0000","last_clean":"2024-03-30T13:01:55.553869+0000","last_became_active":"2024-03-30T13:01:41.983624+0000","last_became_peered":"2024-03-30T13:01:41.983624+0000","last_unstale":"2024-03-30T13:01:55.553869+0000","last_undegraded":"2024-03-30T13:01:55.553869+0000","last_fullsized":"2024-03-30T13:01:55.553869+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:01:51.537497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:51.537497+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:55.057623+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":362,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-30T13:01:52.076482+0000","last_change":"2024-03-30T13:01:42.036177+0000","last_active":"2024-03-30T13:01:52.076482+0000","last_peered":"2024-03-30T13:01:52.076482+0000","last_clean":"2024-03-30T13:01:52.076482+0000","last_became_active":"2024-03-30T13:01:41.986001+0000","last_became_peered":"2024-03-30T13:01:41.986001+0000","last_unstale":"2024-03-30T13:01:52.076482+0000","last_undegraded":"2024-03-30T13:01:52.076482+0000","last_fullsized":"2024-03-30T13:01:52.076482+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:01:27.958371+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:38.392478+0000","scrub_duration":451,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10453822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":247664586,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509519,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48276,"kb_used_data":17088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323564,"statfs":{"total":96636764160,"available":96587329536,"internally_reserved":0,"allocated":17498112,"data_stored":16367989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":156,"seq":670014898182,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53528,"kb_used_data":21832,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94318312,"statfs":{"total":96636764160,"available":96581951488,"internally_reserved":0,"allocated":22355968,"data_stored":21225845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4405,"internal_metadata":32435915},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":135,"seq":579820584976,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45676,"kb_used_data":13536,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94326164,"statfs":{"total":96636764160,"available":96589991936,"internally_reserved":0,"allocated":13860864,"data_stored":12730741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6047,"internal_metadata":32893025},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509520,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39488,"kb_used_data":8304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332352,"statfs":{"total":96636764160,"available":96596328448,"internally_reserved":0,"allocated":8503296,"data_stored":7373173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285605,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38828,"kb_used_data":7136,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94333012,"statfs":{"total":96636764160,"available":96597004288,"internally_reserved":0,"allocated":7307264,"data_stored":6183942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509520,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46676,"kb_used_data":15492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325164,"statfs":{"total":96636764160,"available":96588967936,"internally_reserved":0,"allocated":15863808,"data_stored":14726884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":107,"seq":459561500699,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44140,"kb_used_data":12000,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94327700,"statfs":{"total":96636764160,"available":96591564800,"internally_reserved":0,"allocated":12288000,"data_stored":11171479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6042,"internal_metadata":32893030},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574928,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41484,"kb_used_data":14456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330356,"statfs":{"total":96636764160,"available":96594284544,"internally_reserved":0,"allocated":14802944,"data_stored":13686423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:01:57.491 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:01:57.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:57.604+0000 7f88e47ed640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:57.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:57.616+0000 7f88e47ed640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:58.637 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:01:58.637 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:01:58.659 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2024-03-30T13:01:57.893263+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":247664586,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":358152,"kb_used_data":109904,"kb_used_omap":26,"kb_used_meta":248101,"kb_avail":754616568,"statfs":{"total":773094113280,"available":772727365632,"internally_reserved":0,"allocated":112541696,"data_stored":103493911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27273,"internal_metadata":254055799},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001093"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":401,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-30T13:01:53.149419+0000","last_change":"2024-03-30T13:01:42.155603+0000","last_active":"2024-03-30T13:01:53.149419+0000","last_peered":"2024-03-30T13:01:53.149419+0000","last_clean":"2024-03-30T13:01:53.149419+0000","last_became_active":"2024-03-30T13:01:41.987359+0000","last_became_peered":"2024-03-30T13:01:41.987359+0000","last_unstale":"2024-03-30T13:01:53.149419+0000","last_undegraded":"2024-03-30T13:01:53.149419+0000","last_fullsized":"2024-03-30T13:01:53.149419+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:01:22.928753+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:22.928753+0000","last_clean_scrub_stamp":"2024-03-30T13:01:22.928753+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:51.214009+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18352980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":332,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-30T13:01:54.674496+0000","last_change":"2024-03-30T13:00:58.359297+0000","last_active":"2024-03-30T13:01:54.674496+0000","last_peered":"2024-03-30T13:01:54.674496+0000","last_clean":"2024-03-30T13:01:54.674496+0000","last_became_active":"2024-03-30T13:00:46.110558+0000","last_became_peered":"2024-03-30T13:00:46.110558+0000","last_unstale":"2024-03-30T13:01:54.674496+0000","last_undegraded":"2024-03-30T13:01:54.674496+0000","last_fullsized":"2024-03-30T13:01:54.674496+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:00:58.359040+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:00:58.359040+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:20.772622+0000","scrub_duration":927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":160,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-30T13:01:52.941864+0000","last_change":"2024-03-30T13:01:03.022521+0000","last_active":"2024-03-30T13:01:52.941864+0000","last_peered":"2024-03-30T13:01:52.941864+0000","last_clean":"2024-03-30T13:01:52.941864+0000","last_became_active":"2024-03-30T13:00:46.112427+0000","last_became_peered":"2024-03-30T13:00:46.112427+0000","last_unstale":"2024-03-30T13:01:52.941864+0000","last_undegraded":"2024-03-30T13:01:52.941864+0000","last_fullsized":"2024-03-30T13:01:52.941864+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:01:03.022279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:03.022279+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:08.969141+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":138,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-30T13:01:52.941490+0000","last_change":"2024-03-30T13:00:35.159895+0000","last_active":"2024-03-30T13:01:52.941490+0000","last_peered":"2024-03-30T13:01:52.941490+0000","last_clean":"2024-03-30T13:01:52.941490+0000","last_became_active":"2024-03-30T12:59:55.316713+0000","last_became_peered":"2024-03-30T12:59:55.316713+0000","last_unstale":"2024-03-30T13:01:52.941490+0000","last_undegraded":"2024-03-30T13:01:52.941490+0000","last_fullsized":"2024-03-30T13:01:52.941490+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:00:35.159640+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:00:35.159640+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:04.854390+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":40,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-30T13:01:55.553779+0000","last_change":"2024-03-30T13:01:42.334934+0000","last_active":"2024-03-30T13:01:55.553779+0000","last_peered":"2024-03-30T13:01:55.553779+0000","last_clean":"2024-03-30T13:01:55.553779+0000","last_became_active":"2024-03-30T13:01:41.984313+0000","last_became_peered":"2024-03-30T13:01:41.984313+0000","last_unstale":"2024-03-30T13:01:55.553779+0000","last_undegraded":"2024-03-30T13:01:55.553779+0000","last_fullsized":"2024-03-30T13:01:55.553779+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:01:07.413186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:07.413186+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:35.302199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":399,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-30T13:01:57.635283+0000","last_change":"2024-03-30T13:01:43.093050+0000","last_active":"2024-03-30T13:01:57.635283+0000","last_peered":"2024-03-30T13:01:57.635283+0000","last_clean":"2024-03-30T13:01:57.635283+0000","last_became_active":"2024-03-30T13:01:41.987920+0000","last_became_peered":"2024-03-30T13:01:41.987920+0000","last_unstale":"2024-03-30T13:01:57.635283+0000","last_undegraded":"2024-03-30T13:01:57.635283+0000","last_fullsized":"2024-03-30T13:01:57.635283+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:00:46.414102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:46.414102+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:09.217769+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22789962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":187,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-30T13:01:52.941631+0000","last_change":"2024-03-30T13:01:48.791786+0000","last_active":"2024-03-30T13:01:52.941631+0000","last_peered":"2024-03-30T13:01:52.941631+0000","last_clean":"2024-03-30T13:01:52.941631+0000","last_became_active":"2024-03-30T13:00:46.114131+0000","last_became_peered":"2024-03-30T13:00:46.114131+0000","last_unstale":"2024-03-30T13:01:52.941631+0000","last_undegraded":"2024-03-30T13:01:52.941631+0000","last_fullsized":"2024-03-30T13:01:52.941631+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:01:48.791509+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:01:48.791509+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:04.577057+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":376,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-30T13:01:52.076574+0000","last_change":"2024-03-30T13:01:42.050953+0000","last_active":"2024-03-30T13:01:52.076574+0000","last_peered":"2024-03-30T13:01:52.076574+0000","last_clean":"2024-03-30T13:01:52.076574+0000","last_became_active":"2024-03-30T13:01:41.988151+0000","last_became_peered":"2024-03-30T13:01:41.988151+0000","last_unstale":"2024-03-30T13:01:52.076574+0000","last_undegraded":"2024-03-30T13:01:52.076574+0000","last_fullsized":"2024-03-30T13:01:52.076574+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:01:16.584341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:16.584341+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:35.937908+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":22033680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":428,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-30T13:01:57.635204+0000","last_change":"2024-03-30T13:01:42.116405+0000","last_active":"2024-03-30T13:01:57.635204+0000","last_peered":"2024-03-30T13:01:57.635204+0000","last_clean":"2024-03-30T13:01:57.635204+0000","last_became_active":"2024-03-30T13:01:41.989387+0000","last_became_peered":"2024-03-30T13:01:41.989387+0000","last_unstale":"2024-03-30T13:01:57.635204+0000","last_undegraded":"2024-03-30T13:01:57.635204+0000","last_fullsized":"2024-03-30T13:01:57.635204+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:00:49.391955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:49.391955+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:13.856259+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":30453888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":268,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-30T13:01:54.674344+0000","last_change":"2024-03-30T13:01:13.453369+0000","last_active":"2024-03-30T13:01:54.674344+0000","last_peered":"2024-03-30T13:01:54.674344+0000","last_clean":"2024-03-30T13:01:54.674344+0000","last_became_active":"2024-03-30T13:00:46.109054+0000","last_became_peered":"2024-03-30T13:00:46.109054+0000","last_unstale":"2024-03-30T13:01:54.674344+0000","last_undegraded":"2024-03-30T13:01:54.674344+0000","last_fullsized":"2024-03-30T13:01:54.674344+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:01:13.453092+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:01:13.453092+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:24.209616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":405,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-30T13:01:53.149416+0000","last_change":"2024-03-30T13:01:03.880400+0000","last_active":"2024-03-30T13:01:53.149416+0000","last_peered":"2024-03-30T13:01:53.149416+0000","last_clean":"2024-03-30T13:01:53.149416+0000","last_became_active":"2024-03-30T13:00:46.106211+0000","last_became_peered":"2024-03-30T13:00:46.106211+0000","last_unstale":"2024-03-30T13:01:53.149416+0000","last_undegraded":"2024-03-30T13:01:53.149416+0000","last_fullsized":"2024-03-30T13:01:53.149416+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:01:03.880142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:03.880142+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:16.907695+0000","scrub_duration":251,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25479082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":413,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-03-30T13:01:53.149445+0000","last_change":"2024-03-30T13:01:43.224020+0000","last_active":"2024-03-30T13:01:53.149445+0000","last_peered":"2024-03-30T13:01:53.149445+0000","last_clean":"2024-03-30T13:01:53.149445+0000","last_became_active":"2024-03-30T13:01:41.991185+0000","last_became_peered":"2024-03-30T13:01:41.991185+0000","last_unstale":"2024-03-30T13:01:53.149445+0000","last_undegraded":"2024-03-30T13:01:53.149445+0000","last_fullsized":"2024-03-30T13:01:53.149445+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:00:53.751391+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T13:00:53.751391+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:18.943412+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":54,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-30T13:01:55.553792+0000","last_change":"2024-03-30T13:01:53.533883+0000","last_active":"2024-03-30T13:01:55.553792+0000","last_peered":"2024-03-30T13:01:55.553792+0000","last_clean":"2024-03-30T13:01:55.553792+0000","last_became_active":"2024-03-30T13:01:41.982566+0000","last_became_peered":"2024-03-30T13:01:41.982566+0000","last_unstale":"2024-03-30T13:01:55.553792+0000","last_undegraded":"2024-03-30T13:01:55.553792+0000","last_fullsized":"2024-03-30T13:01:55.553792+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:01:53.533647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:53.533647+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:00.040107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":136,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-30T13:01:52.941492+0000","last_change":"2024-03-30T13:00:39.088475+0000","last_active":"2024-03-30T13:01:52.941492+0000","last_peered":"2024-03-30T13:01:52.941492+0000","last_clean":"2024-03-30T13:01:52.941492+0000","last_became_active":"2024-03-30T12:59:55.310542+0000","last_became_peered":"2024-03-30T12:59:55.310542+0000","last_unstale":"2024-03-30T13:01:52.941492+0000","last_undegraded":"2024-03-30T13:01:52.941492+0000","last_fullsized":"2024-03-30T13:01:52.941492+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:00:39.088323+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:00:39.088323+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:05.618294+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":54,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-30T13:01:55.553869+0000","last_change":"2024-03-30T13:01:51.537905+0000","last_active":"2024-03-30T13:01:55.553869+0000","last_peered":"2024-03-30T13:01:55.553869+0000","last_clean":"2024-03-30T13:01:55.553869+0000","last_became_active":"2024-03-30T13:01:41.983624+0000","last_became_peered":"2024-03-30T13:01:41.983624+0000","last_unstale":"2024-03-30T13:01:55.553869+0000","last_undegraded":"2024-03-30T13:01:55.553869+0000","last_fullsized":"2024-03-30T13:01:55.553869+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:01:51.537497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:51.537497+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:55.057623+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":362,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-03-30T13:01:52.076482+0000","last_change":"2024-03-30T13:01:42.036177+0000","last_active":"2024-03-30T13:01:52.076482+0000","last_peered":"2024-03-30T13:01:52.076482+0000","last_clean":"2024-03-30T13:01:52.076482+0000","last_became_active":"2024-03-30T13:01:41.986001+0000","last_became_peered":"2024-03-30T13:01:41.986001+0000","last_unstale":"2024-03-30T13:01:52.076482+0000","last_undegraded":"2024-03-30T13:01:52.076482+0000","last_fullsized":"2024-03-30T13:01:52.076482+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:01:27.958371+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:38.392478+0000","scrub_duration":451,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10453822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":247664586,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509519,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48276,"kb_used_data":17088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323564,"statfs":{"total":96636764160,"available":96587329536,"internally_reserved":0,"allocated":17498112,"data_stored":16367989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":156,"seq":670014898182,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53528,"kb_used_data":21832,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94318312,"statfs":{"total":96636764160,"available":96581951488,"internally_reserved":0,"allocated":22355968,"data_stored":21225845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4405,"internal_metadata":32435915},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":135,"seq":579820584976,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45676,"kb_used_data":13536,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94326164,"statfs":{"total":96636764160,"available":96589991936,"internally_reserved":0,"allocated":13860864,"data_stored":12730741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6047,"internal_metadata":32893025},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509520,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39488,"kb_used_data":8304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332352,"statfs":{"total":96636764160,"available":96596328448,"internally_reserved":0,"allocated":8503296,"data_stored":7373173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285605,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38828,"kb_used_data":7136,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94333012,"statfs":{"total":96636764160,"available":96597004288,"internally_reserved":0,"allocated":7307264,"data_stored":6183942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509520,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46676,"kb_used_data":15492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325164,"statfs":{"total":96636764160,"available":96588967936,"internally_reserved":0,"allocated":15863808,"data_stored":14726884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":107,"seq":459561500699,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44140,"kb_used_data":12000,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94327700,"statfs":{"total":96636764160,"available":96591564800,"internally_reserved":0,"allocated":12288000,"data_stored":11171479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6042,"internal_metadata":32893030},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574929,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41540,"kb_used_data":14516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330300,"statfs":{"total":96636764160,"available":96594227200,"internally_reserved":0,"allocated":14864384,"data_stored":13713858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:01:58.660 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:01:58.661 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:01:58.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:58.776+0000 7fc2ddfe7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:58.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:58.788+0000 7fc2ddfe7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:58.964 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:01:58.964 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":173,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:01:57.126146+0000","last_up_change":"2024-03-30T13:01:37.322773+0000","last_in_change":"2024-03-30T13:01:40.942619+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":12,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":77,"up_from":83,"up_thru":157,"down_at":78,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6818","nonce":3104224893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6821","nonce":3104224893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6828","nonce":3104224893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6825","nonce":3104224893}]},"public_addr":"172.21.15.67:6818/3104224893","cluster_addr":"172.21.15.67:6821/3104224893","heartbeat_back_addr":"172.21.15.67:6828/3104224893","heartbeat_front_addr":"172.21.15.67:6825/3104224893","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6801","nonce":1504348724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6803","nonce":1504348724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6807","nonce":1504348724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6805","nonce":1504348724}]},"public_addr":"172.21.15.67:6801/1504348724","cluster_addr":"172.21.15.67:6803/1504348724","heartbeat_back_addr":"172.21.15.67:6807/1504348724","heartbeat_front_addr":"172.21.15.67:6805/1504348724","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":36,"last_clean_end":125,"up_from":135,"up_thru":153,"down_at":126,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":2818291404},{"type":"v1","addr":"172.21.15.82:6803","nonce":2818291404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":2818291404},{"type":"v1","addr":"172.21.15.82:6807","nonce":2818291404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":2818291404},{"type":"v1","addr":"172.21.15.82:6815","nonce":2818291404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":2818291404},{"type":"v1","addr":"172.21.15.82:6811","nonce":2818291404}]},"public_addr":"172.21.15.82:6803/2818291404","cluster_addr":"172.21.15.82:6807/2818291404","heartbeat_back_addr":"172.21.15.82:6815/2818291404","heartbeat_front_addr":"172.21.15.82:6811/2818291404","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":151,"up_from":156,"up_thru":157,"down_at":152,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":3428934162},{"type":"v1","addr":"172.21.15.82:6817","nonce":3428934162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":3428934162},{"type":"v1","addr":"172.21.15.82:6819","nonce":3428934162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":3428934162},{"type":"v1","addr":"172.21.15.82:6823","nonce":3428934162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":3428934162},{"type":"v1","addr":"172.21.15.82:6821","nonce":3428934162}]},"public_addr":"172.21.15.82:6817/3428934162","cluster_addr":"172.21.15.82:6819/3428934162","heartbeat_back_addr":"172.21.15.82:6823/3428934162","heartbeat_front_addr":"172.21.15.82:6821/3428934162","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6801","nonce":968508548}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6806","nonce":968508548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6814","nonce":968508548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6810","nonce":968508548}]},"public_addr":"172.21.15.82:6801/968508548","cluster_addr":"172.21.15.82:6806/968508548","heartbeat_back_addr":"172.21.15.82:6814/968508548","heartbeat_front_addr":"172.21.15.82:6810/968508548","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":59,"last_clean_end":100,"up_from":107,"up_thru":135,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2568135668},{"type":"v1","addr":"172.21.15.82:6825","nonce":2568135668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2568135668},{"type":"v1","addr":"172.21.15.82:6827","nonce":2568135668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2568135668},{"type":"v1","addr":"172.21.15.82:6831","nonce":2568135668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2568135668},{"type":"v1","addr":"172.21.15.82:6829","nonce":2568135668}]},"public_addr":"172.21.15.82:6825/2568135668","cluster_addr":"172.21.15.82:6827/2568135668","heartbeat_back_addr":"172.21.15.82:6831/2568135668","heartbeat_front_addr":"172.21.15.82:6829/2568135668","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:01:45.024311+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:01:45.724887+0000","dead_epoch":78},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:01:44.194796+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:00:40.637135+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:00:58.601253+0000","dead_epoch":126},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:01:41.431935+0000","dead_epoch":152},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:01:40.144896+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:00:58.181517+0000","dead_epoch":101}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:01:58.982 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:01:58.983 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:01:59.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:59.084+0000 7fcf3e8f4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:59.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:59.100+0000 7fcf3e8f4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:59.278 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:01:59.297 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:01:59.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:59.416+0000 7f93ff653640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:59.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:59.436+0000 7f93ff653640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:59.615 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:01:59.615 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:01:59.634 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:01:59.634 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:01:59.634 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:01:59.634 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:01:59.634 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 3, 6, 1, 7, 4, 5] 2024-03-30T13:01:59.634 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:01:59.635 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:01:59.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:59.748+0000 7f87c80a9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:59.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:01:59.760+0000 7f87c80a9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:01:59.983 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:01:59.983 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:02:00.000 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-03-30T13:01:59.893773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":247664586,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":358416,"kb_used_data":110168,"kb_used_omap":26,"kb_used_meta":248101,"kb_avail":754616304,"statfs":{"total":773094113280,"available":772727095296,"internally_reserved":0,"allocated":112812032,"data_stored":103614625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27273,"internal_metadata":254055799},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.783094"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":408,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-30T13:01:57.993687+0000","last_change":"2024-03-30T13:01:42.155603+0000","last_active":"2024-03-30T13:01:57.993687+0000","last_peered":"2024-03-30T13:01:57.993687+0000","last_clean":"2024-03-30T13:01:57.993687+0000","last_became_active":"2024-03-30T13:01:41.987359+0000","last_became_peered":"2024-03-30T13:01:41.987359+0000","last_unstale":"2024-03-30T13:01:57.993687+0000","last_undegraded":"2024-03-30T13:01:57.993687+0000","last_fullsized":"2024-03-30T13:01:57.993687+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:01:22.928753+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:22.928753+0000","last_clean_scrub_stamp":"2024-03-30T13:01:22.928753+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:51.214009+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18352980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":332,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-30T13:01:54.674496+0000","last_change":"2024-03-30T13:00:58.359297+0000","last_active":"2024-03-30T13:01:54.674496+0000","last_peered":"2024-03-30T13:01:54.674496+0000","last_clean":"2024-03-30T13:01:54.674496+0000","last_became_active":"2024-03-30T13:00:46.110558+0000","last_became_peered":"2024-03-30T13:00:46.110558+0000","last_unstale":"2024-03-30T13:01:54.674496+0000","last_undegraded":"2024-03-30T13:01:54.674496+0000","last_fullsized":"2024-03-30T13:01:54.674496+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:00:58.359040+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:00:58.359040+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:20.772622+0000","scrub_duration":927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":169,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-30T13:01:57.134656+0000","last_change":"2024-03-30T13:01:03.022521+0000","last_active":"2024-03-30T13:01:57.134656+0000","last_peered":"2024-03-30T13:01:57.134656+0000","last_clean":"2024-03-30T13:01:57.134656+0000","last_became_active":"2024-03-30T13:00:46.112427+0000","last_became_peered":"2024-03-30T13:00:46.112427+0000","last_unstale":"2024-03-30T13:01:57.134656+0000","last_undegraded":"2024-03-30T13:01:57.134656+0000","last_fullsized":"2024-03-30T13:01:57.134656+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:01:03.022279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:03.022279+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:08.969141+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":148,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-30T13:01:57.134911+0000","last_change":"2024-03-30T13:00:35.159895+0000","last_active":"2024-03-30T13:01:57.134911+0000","last_peered":"2024-03-30T13:01:57.134911+0000","last_clean":"2024-03-30T13:01:57.134911+0000","last_became_active":"2024-03-30T12:59:55.316713+0000","last_became_peered":"2024-03-30T12:59:55.316713+0000","last_unstale":"2024-03-30T13:01:57.134911+0000","last_undegraded":"2024-03-30T13:01:57.134911+0000","last_fullsized":"2024-03-30T13:01:57.134911+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:00:35.159640+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:00:35.159640+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:04.854390+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":40,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-30T13:01:55.553779+0000","last_change":"2024-03-30T13:01:42.334934+0000","last_active":"2024-03-30T13:01:55.553779+0000","last_peered":"2024-03-30T13:01:55.553779+0000","last_clean":"2024-03-30T13:01:55.553779+0000","last_became_active":"2024-03-30T13:01:41.984313+0000","last_became_peered":"2024-03-30T13:01:41.984313+0000","last_unstale":"2024-03-30T13:01:55.553779+0000","last_undegraded":"2024-03-30T13:01:55.553779+0000","last_fullsized":"2024-03-30T13:01:55.553779+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:01:07.413186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:07.413186+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:35.302199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":399,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-30T13:01:57.635283+0000","last_change":"2024-03-30T13:01:43.093050+0000","last_active":"2024-03-30T13:01:57.635283+0000","last_peered":"2024-03-30T13:01:57.635283+0000","last_clean":"2024-03-30T13:01:57.635283+0000","last_became_active":"2024-03-30T13:01:41.987920+0000","last_became_peered":"2024-03-30T13:01:41.987920+0000","last_unstale":"2024-03-30T13:01:57.635283+0000","last_undegraded":"2024-03-30T13:01:57.635283+0000","last_fullsized":"2024-03-30T13:01:57.635283+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:00:46.414102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:46.414102+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:09.217769+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22789962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":196,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-30T13:01:57.134592+0000","last_change":"2024-03-30T13:01:48.791786+0000","last_active":"2024-03-30T13:01:57.134592+0000","last_peered":"2024-03-30T13:01:57.134592+0000","last_clean":"2024-03-30T13:01:57.134592+0000","last_became_active":"2024-03-30T13:00:46.114131+0000","last_became_peered":"2024-03-30T13:00:46.114131+0000","last_unstale":"2024-03-30T13:01:57.134592+0000","last_undegraded":"2024-03-30T13:01:57.134592+0000","last_fullsized":"2024-03-30T13:01:57.134592+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:01:48.791509+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:01:48.791509+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:04.577057+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":386,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-30T13:01:57.131972+0000","last_change":"2024-03-30T13:01:42.050953+0000","last_active":"2024-03-30T13:01:57.131972+0000","last_peered":"2024-03-30T13:01:57.131972+0000","last_clean":"2024-03-30T13:01:57.131972+0000","last_became_active":"2024-03-30T13:01:41.988151+0000","last_became_peered":"2024-03-30T13:01:41.988151+0000","last_unstale":"2024-03-30T13:01:57.131972+0000","last_undegraded":"2024-03-30T13:01:57.131972+0000","last_fullsized":"2024-03-30T13:01:57.131972+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:01:16.584341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:16.584341+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:35.937908+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":22033680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":428,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-30T13:01:57.635204+0000","last_change":"2024-03-30T13:01:42.116405+0000","last_active":"2024-03-30T13:01:57.635204+0000","last_peered":"2024-03-30T13:01:57.635204+0000","last_clean":"2024-03-30T13:01:57.635204+0000","last_became_active":"2024-03-30T13:01:41.989387+0000","last_became_peered":"2024-03-30T13:01:41.989387+0000","last_unstale":"2024-03-30T13:01:57.635204+0000","last_undegraded":"2024-03-30T13:01:57.635204+0000","last_fullsized":"2024-03-30T13:01:57.635204+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:00:49.391955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:49.391955+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:13.856259+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":30453888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":268,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-03-30T13:01:54.674344+0000","last_change":"2024-03-30T13:01:13.453369+0000","last_active":"2024-03-30T13:01:54.674344+0000","last_peered":"2024-03-30T13:01:54.674344+0000","last_clean":"2024-03-30T13:01:54.674344+0000","last_became_active":"2024-03-30T13:00:46.109054+0000","last_became_peered":"2024-03-30T13:00:46.109054+0000","last_unstale":"2024-03-30T13:01:54.674344+0000","last_undegraded":"2024-03-30T13:01:54.674344+0000","last_fullsized":"2024-03-30T13:01:54.674344+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:01:13.453092+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:01:13.453092+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:24.209616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":412,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-30T13:01:57.993627+0000","last_change":"2024-03-30T13:01:03.880400+0000","last_active":"2024-03-30T13:01:57.993627+0000","last_peered":"2024-03-30T13:01:57.993627+0000","last_clean":"2024-03-30T13:01:57.993627+0000","last_became_active":"2024-03-30T13:00:46.106211+0000","last_became_peered":"2024-03-30T13:00:46.106211+0000","last_unstale":"2024-03-30T13:01:57.993627+0000","last_undegraded":"2024-03-30T13:01:57.993627+0000","last_fullsized":"2024-03-30T13:01:57.993627+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:01:03.880142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:03.880142+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:16.907695+0000","scrub_duration":251,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25479082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":420,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-30T13:01:57.993440+0000","last_change":"2024-03-30T13:01:43.224020+0000","last_active":"2024-03-30T13:01:57.993440+0000","last_peered":"2024-03-30T13:01:57.993440+0000","last_clean":"2024-03-30T13:01:57.993440+0000","last_became_active":"2024-03-30T13:01:41.991185+0000","last_became_peered":"2024-03-30T13:01:41.991185+0000","last_unstale":"2024-03-30T13:01:57.993440+0000","last_undegraded":"2024-03-30T13:01:57.993440+0000","last_fullsized":"2024-03-30T13:01:57.993440+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:00:53.751391+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T13:00:53.751391+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:18.943412+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":54,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-30T13:01:55.553792+0000","last_change":"2024-03-30T13:01:53.533883+0000","last_active":"2024-03-30T13:01:55.553792+0000","last_peered":"2024-03-30T13:01:55.553792+0000","last_clean":"2024-03-30T13:01:55.553792+0000","last_became_active":"2024-03-30T13:01:41.982566+0000","last_became_peered":"2024-03-30T13:01:41.982566+0000","last_unstale":"2024-03-30T13:01:55.553792+0000","last_undegraded":"2024-03-30T13:01:55.553792+0000","last_fullsized":"2024-03-30T13:01:55.553792+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:01:53.533647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:53.533647+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:00.040107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":145,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-30T13:01:57.134543+0000","last_change":"2024-03-30T13:00:39.088475+0000","last_active":"2024-03-30T13:01:57.134543+0000","last_peered":"2024-03-30T13:01:57.134543+0000","last_clean":"2024-03-30T13:01:57.134543+0000","last_became_active":"2024-03-30T12:59:55.310542+0000","last_became_peered":"2024-03-30T12:59:55.310542+0000","last_unstale":"2024-03-30T13:01:57.134543+0000","last_undegraded":"2024-03-30T13:01:57.134543+0000","last_fullsized":"2024-03-30T13:01:57.134543+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:00:39.088323+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:00:39.088323+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:05.618294+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":54,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-03-30T13:01:55.553869+0000","last_change":"2024-03-30T13:01:51.537905+0000","last_active":"2024-03-30T13:01:55.553869+0000","last_peered":"2024-03-30T13:01:55.553869+0000","last_clean":"2024-03-30T13:01:55.553869+0000","last_became_active":"2024-03-30T13:01:41.983624+0000","last_became_peered":"2024-03-30T13:01:41.983624+0000","last_unstale":"2024-03-30T13:01:55.553869+0000","last_undegraded":"2024-03-30T13:01:55.553869+0000","last_fullsized":"2024-03-30T13:01:55.553869+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:01:51.537497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:51.537497+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:55.057623+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":372,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-30T13:01:57.131972+0000","last_change":"2024-03-30T13:01:42.036177+0000","last_active":"2024-03-30T13:01:57.131972+0000","last_peered":"2024-03-30T13:01:57.131972+0000","last_clean":"2024-03-30T13:01:57.131972+0000","last_became_active":"2024-03-30T13:01:41.986001+0000","last_became_peered":"2024-03-30T13:01:41.986001+0000","last_unstale":"2024-03-30T13:01:57.131972+0000","last_undegraded":"2024-03-30T13:01:57.131972+0000","last_fullsized":"2024-03-30T13:01:57.131972+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:01:27.958371+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:38.392478+0000","scrub_duration":451,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10453822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":247664586,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509520,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48316,"kb_used_data":17136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323524,"statfs":{"total":96636764160,"available":96587288576,"internally_reserved":0,"allocated":17547264,"data_stored":16389937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":156,"seq":670014898182,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53528,"kb_used_data":21832,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94318312,"statfs":{"total":96636764160,"available":96581951488,"internally_reserved":0,"allocated":22355968,"data_stored":21225845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4405,"internal_metadata":32435915},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":135,"seq":579820584976,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45676,"kb_used_data":13536,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94326164,"statfs":{"total":96636764160,"available":96589991936,"internally_reserved":0,"allocated":13860864,"data_stored":12730741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6047,"internal_metadata":32893025},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509521,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39540,"kb_used_data":8352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332300,"statfs":{"total":96636764160,"available":96596275200,"internally_reserved":0,"allocated":8552448,"data_stored":7395121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285606,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38896,"kb_used_data":7196,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94332944,"statfs":{"total":96636764160,"available":96596934656,"internally_reserved":0,"allocated":7368704,"data_stored":6211377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509521,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46708,"kb_used_data":15528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325132,"statfs":{"total":96636764160,"available":96588935168,"internally_reserved":0,"allocated":15900672,"data_stored":14743345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":107,"seq":459561500700,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44212,"kb_used_data":12072,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94327628,"statfs":{"total":96636764160,"available":96591491072,"internally_reserved":0,"allocated":12361728,"data_stored":11204401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6042,"internal_metadata":32893030},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574929,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41540,"kb_used_data":14516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330300,"statfs":{"total":96636764160,"available":96594227200,"internally_reserved":0,"allocated":14864384,"data_stored":13713858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:02:00.001 INFO:tasks.thrashosds.thrasher:Choosing PG 1.9 with acting set [7, 3, 2] 2024-03-30T13:02:00.002 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 2, 3, 6, 1, 7, 4, 5] 2024-03-30T13:02:00.002 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-30T13:02:00.002 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:02:00.073 INFO:tasks.ceph.osd.7:Stopped 2024-03-30T13:02:00.074 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 2, 3, 6, 1, 7, 4, 5] 2024-03-30T13:02:00.074 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 7 2024-03-30T13:02:00.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:00.196+0000 7fe1da64d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:00.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:00.212+0000 7fe1da64d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:00.400 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:02:00.396+0000 7fbb38f08640 -1 mon.a@0(leader).osd e174 definitely_dead 0 2024-03-30T13:02:01.174 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.7. 2024-03-30T13:02:01.199 INFO:tasks.thrashosds.thrasher:dead_osds=[7], live_osds=[0, 2, 3, 6, 1, 4, 5] 2024-03-30T13:02:01.199 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:02:01.199 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:02:01.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:01.316+0000 7fd09fde6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:01.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:01.328+0000 7fd09fde6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:01.509 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:02:01.509 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:02:01.527 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-03-30T13:02:01.165132+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":247664586,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":314316,"kb_used_data":98204,"kb_used_omap":20,"kb_used_meta":215979,"kb_avail":660288564,"statfs":{"total":676457349120,"available":676135489536,"internally_reserved":0,"allocated":100560896,"data_stored":92459783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21231,"internal_metadata":221162769},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.272428"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":408,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-30T13:01:57.993687+0000","last_change":"2024-03-30T13:01:42.155603+0000","last_active":"2024-03-30T13:01:57.993687+0000","last_peered":"2024-03-30T13:01:57.993687+0000","last_clean":"2024-03-30T13:01:57.993687+0000","last_became_active":"2024-03-30T13:01:41.987359+0000","last_became_peered":"2024-03-30T13:01:41.987359+0000","last_unstale":"2024-03-30T13:01:57.993687+0000","last_undegraded":"2024-03-30T13:01:57.993687+0000","last_fullsized":"2024-03-30T13:01:57.993687+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:01:22.928753+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:22.928753+0000","last_clean_scrub_stamp":"2024-03-30T13:01:22.928753+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:51.214009+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18352980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":338,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-30T13:01:58.441526+0000","last_change":"2024-03-30T13:00:58.359297+0000","last_active":"2024-03-30T13:01:58.441526+0000","last_peered":"2024-03-30T13:01:58.441526+0000","last_clean":"2024-03-30T13:01:58.441526+0000","last_became_active":"2024-03-30T13:00:46.110558+0000","last_became_peered":"2024-03-30T13:00:46.110558+0000","last_unstale":"2024-03-30T13:01:58.441526+0000","last_undegraded":"2024-03-30T13:01:58.441526+0000","last_fullsized":"2024-03-30T13:01:58.441526+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:00:58.359040+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:00:58.359040+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:20.772622+0000","scrub_duration":927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":169,"reported_epoch":173,"state":"stale+active+clean","last_fresh":"2024-03-30T13:01:57.134656+0000","last_change":"2024-03-30T13:01:03.022521+0000","last_active":"2024-03-30T13:01:57.134656+0000","last_peered":"2024-03-30T13:01:57.134656+0000","last_clean":"2024-03-30T13:01:57.134656+0000","last_became_active":"2024-03-30T13:00:46.112427+0000","last_became_peered":"2024-03-30T13:00:46.112427+0000","last_unstale":"2024-03-30T13:02:00.155349+0000","last_undegraded":"2024-03-30T13:01:57.134656+0000","last_fullsized":"2024-03-30T13:01:57.134656+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:01:03.022279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:03.022279+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:08.969141+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":148,"reported_epoch":173,"state":"stale+active+clean","last_fresh":"2024-03-30T13:01:57.134911+0000","last_change":"2024-03-30T13:00:35.159895+0000","last_active":"2024-03-30T13:01:57.134911+0000","last_peered":"2024-03-30T13:01:57.134911+0000","last_clean":"2024-03-30T13:01:57.134911+0000","last_became_active":"2024-03-30T12:59:55.316713+0000","last_became_peered":"2024-03-30T12:59:55.316713+0000","last_unstale":"2024-03-30T13:02:00.155351+0000","last_undegraded":"2024-03-30T13:01:57.134911+0000","last_fullsized":"2024-03-30T13:01:57.134911+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:00:35.159640+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:00:35.159640+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:04.854390+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":46,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-30T13:02:00.166803+0000","last_change":"2024-03-30T13:01:42.334934+0000","last_active":"2024-03-30T13:02:00.166803+0000","last_peered":"2024-03-30T13:02:00.166803+0000","last_clean":"2024-03-30T13:02:00.166803+0000","last_became_active":"2024-03-30T13:01:41.984313+0000","last_became_peered":"2024-03-30T13:01:41.984313+0000","last_unstale":"2024-03-30T13:02:00.166803+0000","last_undegraded":"2024-03-30T13:02:00.166803+0000","last_fullsized":"2024-03-30T13:02:00.166803+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:01:07.413186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:07.413186+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:35.302199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":399,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-30T13:01:57.635283+0000","last_change":"2024-03-30T13:01:43.093050+0000","last_active":"2024-03-30T13:01:57.635283+0000","last_peered":"2024-03-30T13:01:57.635283+0000","last_clean":"2024-03-30T13:01:57.635283+0000","last_became_active":"2024-03-30T13:01:41.987920+0000","last_became_peered":"2024-03-30T13:01:41.987920+0000","last_unstale":"2024-03-30T13:01:57.635283+0000","last_undegraded":"2024-03-30T13:01:57.635283+0000","last_fullsized":"2024-03-30T13:01:57.635283+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:00:46.414102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:46.414102+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:09.217769+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22789962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":196,"reported_epoch":173,"state":"stale+active+clean","last_fresh":"2024-03-30T13:01:57.134592+0000","last_change":"2024-03-30T13:01:48.791786+0000","last_active":"2024-03-30T13:01:57.134592+0000","last_peered":"2024-03-30T13:01:57.134592+0000","last_clean":"2024-03-30T13:01:57.134592+0000","last_became_active":"2024-03-30T13:00:46.114131+0000","last_became_peered":"2024-03-30T13:00:46.114131+0000","last_unstale":"2024-03-30T13:02:00.155353+0000","last_undegraded":"2024-03-30T13:01:57.134592+0000","last_fullsized":"2024-03-30T13:01:57.134592+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:01:48.791509+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:01:48.791509+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:04.577057+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":386,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-30T13:01:57.131972+0000","last_change":"2024-03-30T13:01:42.050953+0000","last_active":"2024-03-30T13:01:57.131972+0000","last_peered":"2024-03-30T13:01:57.131972+0000","last_clean":"2024-03-30T13:01:57.131972+0000","last_became_active":"2024-03-30T13:01:41.988151+0000","last_became_peered":"2024-03-30T13:01:41.988151+0000","last_unstale":"2024-03-30T13:01:57.131972+0000","last_undegraded":"2024-03-30T13:01:57.131972+0000","last_fullsized":"2024-03-30T13:01:57.131972+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:01:16.584341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:16.584341+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:35.937908+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":22033680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":428,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-30T13:01:57.635204+0000","last_change":"2024-03-30T13:01:42.116405+0000","last_active":"2024-03-30T13:01:57.635204+0000","last_peered":"2024-03-30T13:01:57.635204+0000","last_clean":"2024-03-30T13:01:57.635204+0000","last_became_active":"2024-03-30T13:01:41.989387+0000","last_became_peered":"2024-03-30T13:01:41.989387+0000","last_unstale":"2024-03-30T13:01:57.635204+0000","last_undegraded":"2024-03-30T13:01:57.635204+0000","last_fullsized":"2024-03-30T13:01:57.635204+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:00:49.391955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:00:49.391955+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:13.856259+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":30453888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":274,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-30T13:01:58.441527+0000","last_change":"2024-03-30T13:01:13.453369+0000","last_active":"2024-03-30T13:01:58.441527+0000","last_peered":"2024-03-30T13:01:58.441527+0000","last_clean":"2024-03-30T13:01:58.441527+0000","last_became_active":"2024-03-30T13:00:46.109054+0000","last_became_peered":"2024-03-30T13:00:46.109054+0000","last_unstale":"2024-03-30T13:01:58.441527+0000","last_undegraded":"2024-03-30T13:01:58.441527+0000","last_fullsized":"2024-03-30T13:01:58.441527+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:01:13.453092+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:01:13.453092+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:24.209616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":412,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-30T13:01:57.993627+0000","last_change":"2024-03-30T13:01:03.880400+0000","last_active":"2024-03-30T13:01:57.993627+0000","last_peered":"2024-03-30T13:01:57.993627+0000","last_clean":"2024-03-30T13:01:57.993627+0000","last_became_active":"2024-03-30T13:00:46.106211+0000","last_became_peered":"2024-03-30T13:00:46.106211+0000","last_unstale":"2024-03-30T13:01:57.993627+0000","last_undegraded":"2024-03-30T13:01:57.993627+0000","last_fullsized":"2024-03-30T13:01:57.993627+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:01:03.880142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:03.880142+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:16.907695+0000","scrub_duration":251,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25479082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":420,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-30T13:01:57.993440+0000","last_change":"2024-03-30T13:01:43.224020+0000","last_active":"2024-03-30T13:01:57.993440+0000","last_peered":"2024-03-30T13:01:57.993440+0000","last_clean":"2024-03-30T13:01:57.993440+0000","last_became_active":"2024-03-30T13:01:41.991185+0000","last_became_peered":"2024-03-30T13:01:41.991185+0000","last_unstale":"2024-03-30T13:01:57.993440+0000","last_undegraded":"2024-03-30T13:01:57.993440+0000","last_fullsized":"2024-03-30T13:01:57.993440+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:00:53.751391+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T13:00:53.751391+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:18.943412+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":60,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-30T13:02:00.166791+0000","last_change":"2024-03-30T13:01:53.533883+0000","last_active":"2024-03-30T13:02:00.166791+0000","last_peered":"2024-03-30T13:02:00.166791+0000","last_clean":"2024-03-30T13:02:00.166791+0000","last_became_active":"2024-03-30T13:01:41.982566+0000","last_became_peered":"2024-03-30T13:01:41.982566+0000","last_unstale":"2024-03-30T13:02:00.166791+0000","last_undegraded":"2024-03-30T13:02:00.166791+0000","last_fullsized":"2024-03-30T13:02:00.166791+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:01:53.533647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:53.533647+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:00.040107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":145,"reported_epoch":173,"state":"stale+active+clean","last_fresh":"2024-03-30T13:01:57.134543+0000","last_change":"2024-03-30T13:00:39.088475+0000","last_active":"2024-03-30T13:01:57.134543+0000","last_peered":"2024-03-30T13:01:57.134543+0000","last_clean":"2024-03-30T13:01:57.134543+0000","last_became_active":"2024-03-30T12:59:55.310542+0000","last_became_peered":"2024-03-30T12:59:55.310542+0000","last_unstale":"2024-03-30T13:02:00.155356+0000","last_undegraded":"2024-03-30T13:01:57.134543+0000","last_fullsized":"2024-03-30T13:01:57.134543+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:00:39.088323+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:00:39.088323+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:05.618294+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":60,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-30T13:02:00.166945+0000","last_change":"2024-03-30T13:01:51.537905+0000","last_active":"2024-03-30T13:02:00.166945+0000","last_peered":"2024-03-30T13:02:00.166945+0000","last_clean":"2024-03-30T13:02:00.166945+0000","last_became_active":"2024-03-30T13:01:41.983624+0000","last_became_peered":"2024-03-30T13:01:41.983624+0000","last_unstale":"2024-03-30T13:02:00.166945+0000","last_undegraded":"2024-03-30T13:02:00.166945+0000","last_fullsized":"2024-03-30T13:02:00.166945+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:01:51.537497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:51.537497+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:55.057623+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":372,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-30T13:01:57.131972+0000","last_change":"2024-03-30T13:01:42.036177+0000","last_active":"2024-03-30T13:01:57.131972+0000","last_peered":"2024-03-30T13:01:57.131972+0000","last_clean":"2024-03-30T13:01:57.131972+0000","last_became_active":"2024-03-30T13:01:41.986001+0000","last_became_peered":"2024-03-30T13:01:41.986001+0000","last_unstale":"2024-03-30T13:01:57.131972+0000","last_undegraded":"2024-03-30T13:01:57.131972+0000","last_fullsized":"2024-03-30T13:01:57.131972+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:01:27.958371+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:38.392478+0000","scrub_duration":451,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10453822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":247664586,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509520,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48316,"kb_used_data":17136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323524,"statfs":{"total":96636764160,"available":96587288576,"internally_reserved":0,"allocated":17547264,"data_stored":16389937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":156,"seq":670014898183,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53588,"kb_used_data":21892,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94318252,"statfs":{"total":96636764160,"available":96581890048,"internally_reserved":0,"allocated":22417408,"data_stored":21253456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4405,"internal_metadata":32435915},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":135,"seq":579820584977,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45728,"kb_used_data":13584,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94326112,"statfs":{"total":96636764160,"available":96589938688,"internally_reserved":0,"allocated":13910016,"data_stored":12752689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6047,"internal_metadata":32893025},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509521,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39540,"kb_used_data":8352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332300,"statfs":{"total":96636764160,"available":96596275200,"internally_reserved":0,"allocated":8552448,"data_stored":7395121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285606,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38896,"kb_used_data":7196,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94332944,"statfs":{"total":96636764160,"available":96596934656,"internally_reserved":0,"allocated":7368704,"data_stored":6211377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509521,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46708,"kb_used_data":15528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325132,"statfs":{"total":96636764160,"available":96588935168,"internally_reserved":0,"allocated":15900672,"data_stored":14743345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574929,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41540,"kb_used_data":14516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330300,"statfs":{"total":96636764160,"available":96594227200,"internally_reserved":0,"allocated":14864384,"data_stored":13713858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:02:01.528 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:02:26.529 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:02:26.529 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:02:26.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:26.652+0000 7f9fa9d0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:26.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:26.664+0000 7f9fa9d0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:26.844 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:02:26.845 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:02:26.863 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-03-30T13:02:25.900821+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":232941852,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":324748,"kb_used_data":108668,"kb_used_omap":20,"kb_used_meta":215979,"kb_avail":660278132,"statfs":{"total":676457349120,"available":676124807168,"internally_reserved":0,"allocated":111276032,"data_stored":102939270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21231,"internal_metadata":221162769},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003248"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":418,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-30T13:02:13.952994+0000","last_change":"2024-03-30T13:01:42.155603+0000","last_active":"2024-03-30T13:02:13.952994+0000","last_peered":"2024-03-30T13:02:13.952994+0000","last_clean":"2024-03-30T13:02:13.952994+0000","last_became_active":"2024-03-30T13:01:41.987359+0000","last_became_peered":"2024-03-30T13:01:41.987359+0000","last_unstale":"2024-03-30T13:02:13.952994+0000","last_undegraded":"2024-03-30T13:02:13.952994+0000","last_fullsized":"2024-03-30T13:02:13.952994+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:01:22.928753+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:22.928753+0000","last_clean_scrub_stamp":"2024-03-30T13:01:22.928753+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:51.214009+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18352980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":357,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-30T13:02:13.227066+0000","last_change":"2024-03-30T13:02:02.223991+0000","last_active":"2024-03-30T13:02:13.227066+0000","last_peered":"2024-03-30T13:02:13.227066+0000","last_clean":"2024-03-30T13:02:13.227066+0000","last_became_active":"2024-03-30T13:02:02.189445+0000","last_became_peered":"2024-03-30T13:02:02.189445+0000","last_unstale":"2024-03-30T13:02:13.227066+0000","last_undegraded":"2024-03-30T13:02:13.227066+0000","last_fullsized":"2024-03-30T13:02:13.227066+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:00:58.359040+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:00:58.359040+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:21.078259+0000","scrub_duration":927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":23,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-30T13:02:13.228407+0000","last_change":"2024-03-30T13:02:02.268864+0000","last_active":"2024-03-30T13:02:13.228407+0000","last_peered":"2024-03-30T13:02:13.228407+0000","last_clean":"2024-03-30T13:02:13.228407+0000","last_became_active":"2024-03-30T13:02:02.184517+0000","last_became_peered":"2024-03-30T13:02:02.184517+0000","last_unstale":"2024-03-30T13:02:13.228407+0000","last_undegraded":"2024-03-30T13:02:13.228407+0000","last_fullsized":"2024-03-30T13:02:13.228407+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:01:03.022279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:03.022279+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:29.970206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":29,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-30T13:02:13.226588+0000","last_change":"2024-03-30T13:02:02.360190+0000","last_active":"2024-03-30T13:02:13.226588+0000","last_peered":"2024-03-30T13:02:13.226588+0000","last_clean":"2024-03-30T13:02:13.226588+0000","last_became_active":"2024-03-30T13:02:02.187356+0000","last_became_peered":"2024-03-30T13:02:02.187356+0000","last_unstale":"2024-03-30T13:02:13.226588+0000","last_undegraded":"2024-03-30T13:02:13.226588+0000","last_fullsized":"2024-03-30T13:02:13.226588+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:02:02.359931+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:02:02.359931+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:14.299077+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":54,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-30T13:02:13.952179+0000","last_change":"2024-03-30T13:01:42.334934+0000","last_active":"2024-03-30T13:02:13.952179+0000","last_peered":"2024-03-30T13:02:13.952179+0000","last_clean":"2024-03-30T13:02:13.952179+0000","last_became_active":"2024-03-30T13:01:41.984313+0000","last_became_peered":"2024-03-30T13:01:41.984313+0000","last_unstale":"2024-03-30T13:02:13.952179+0000","last_undegraded":"2024-03-30T13:02:13.952179+0000","last_fullsized":"2024-03-30T13:02:13.952179+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:01:07.413186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:07.413186+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:35.302199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":417,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-30T13:02:13.952244+0000","last_change":"2024-03-30T13:02:10.596835+0000","last_active":"2024-03-30T13:02:13.952244+0000","last_peered":"2024-03-30T13:02:13.952244+0000","last_clean":"2024-03-30T13:02:13.952244+0000","last_became_active":"2024-03-30T13:01:41.987920+0000","last_became_peered":"2024-03-30T13:01:41.987920+0000","last_unstale":"2024-03-30T13:02:13.952244+0000","last_undegraded":"2024-03-30T13:02:13.952244+0000","last_fullsized":"2024-03-30T13:02:13.952244+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:02:10.596458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:10.596458+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:38.665640+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22789962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":36,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-30T13:02:13.227011+0000","last_change":"2024-03-30T13:02:02.318531+0000","last_active":"2024-03-30T13:02:13.227011+0000","last_peered":"2024-03-30T13:02:13.227011+0000","last_clean":"2024-03-30T13:02:13.227011+0000","last_became_active":"2024-03-30T13:02:02.189796+0000","last_became_peered":"2024-03-30T13:02:02.189796+0000","last_unstale":"2024-03-30T13:02:13.227011+0000","last_undegraded":"2024-03-30T13:02:13.227011+0000","last_fullsized":"2024-03-30T13:02:13.227011+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:01:48.791509+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:01:48.791509+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:01.807918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":396,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-30T13:02:13.226592+0000","last_change":"2024-03-30T13:01:42.050953+0000","last_active":"2024-03-30T13:02:13.226592+0000","last_peered":"2024-03-30T13:02:13.226592+0000","last_clean":"2024-03-30T13:02:13.226592+0000","last_became_active":"2024-03-30T13:01:41.988151+0000","last_became_peered":"2024-03-30T13:01:41.988151+0000","last_unstale":"2024-03-30T13:02:13.226592+0000","last_undegraded":"2024-03-30T13:02:13.226592+0000","last_fullsized":"2024-03-30T13:02:13.226592+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:01:16.584341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:16.584341+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:35.937908+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":22033680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":458,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-30T13:02:13.952456+0000","last_change":"2024-03-30T13:02:08.597804+0000","last_active":"2024-03-30T13:02:13.952456+0000","last_peered":"2024-03-30T13:02:13.952456+0000","last_clean":"2024-03-30T13:02:13.952456+0000","last_became_active":"2024-03-30T13:02:02.189265+0000","last_became_peered":"2024-03-30T13:02:02.189265+0000","last_unstale":"2024-03-30T13:02:13.952456+0000","last_undegraded":"2024-03-30T13:02:13.952456+0000","last_fullsized":"2024-03-30T13:02:13.952456+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:02:08.597557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:08.597557+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:15.421010+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":35529536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":284,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-30T13:02:13.226982+0000","last_change":"2024-03-30T13:01:13.453369+0000","last_active":"2024-03-30T13:02:13.226982+0000","last_peered":"2024-03-30T13:02:13.226982+0000","last_clean":"2024-03-30T13:02:13.226982+0000","last_became_active":"2024-03-30T13:00:46.109054+0000","last_became_peered":"2024-03-30T13:00:46.109054+0000","last_unstale":"2024-03-30T13:02:13.226982+0000","last_undegraded":"2024-03-30T13:02:13.226982+0000","last_fullsized":"2024-03-30T13:02:13.226982+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:01:13.453092+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:01:13.453092+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:24.209616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":431,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-30T13:02:18.933842+0000","last_change":"2024-03-30T13:02:18.934000+0000","last_active":"2024-03-30T13:02:18.933842+0000","last_peered":"2024-03-30T13:02:18.933842+0000","last_clean":"2024-03-30T13:02:18.933842+0000","last_became_active":"2024-03-30T13:00:46.106211+0000","last_became_peered":"2024-03-30T13:00:46.106211+0000","last_unstale":"2024-03-30T13:02:18.933842+0000","last_undegraded":"2024-03-30T13:02:18.933842+0000","last_fullsized":"2024-03-30T13:02:18.933842+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:02:18.933526+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:42.090625+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25479082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":431,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-30T13:02:13.953091+0000","last_change":"2024-03-30T13:01:43.224020+0000","last_active":"2024-03-30T13:02:13.953091+0000","last_peered":"2024-03-30T13:02:13.953091+0000","last_clean":"2024-03-30T13:02:13.953091+0000","last_became_active":"2024-03-30T13:01:41.991185+0000","last_became_peered":"2024-03-30T13:01:41.991185+0000","last_unstale":"2024-03-30T13:02:13.953091+0000","last_undegraded":"2024-03-30T13:02:13.953091+0000","last_fullsized":"2024-03-30T13:02:13.953091+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:00:53.751391+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T13:00:53.751391+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:18.943412+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":68,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-30T13:02:13.952068+0000","last_change":"2024-03-30T13:01:53.533883+0000","last_active":"2024-03-30T13:02:13.952068+0000","last_peered":"2024-03-30T13:02:13.952068+0000","last_clean":"2024-03-30T13:02:13.952068+0000","last_became_active":"2024-03-30T13:01:41.982566+0000","last_became_peered":"2024-03-30T13:01:41.982566+0000","last_unstale":"2024-03-30T13:02:13.952068+0000","last_undegraded":"2024-03-30T13:02:13.952068+0000","last_fullsized":"2024-03-30T13:02:13.952068+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:01:53.533647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:53.533647+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:00.040107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":27,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-30T13:02:13.952063+0000","last_change":"2024-03-30T13:02:08.667129+0000","last_active":"2024-03-30T13:02:13.952063+0000","last_peered":"2024-03-30T13:02:13.952063+0000","last_clean":"2024-03-30T13:02:13.952063+0000","last_became_active":"2024-03-30T13:02:02.187235+0000","last_became_peered":"2024-03-30T13:02:02.187235+0000","last_unstale":"2024-03-30T13:02:13.952063+0000","last_undegraded":"2024-03-30T13:02:13.952063+0000","last_fullsized":"2024-03-30T13:02:13.952063+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:08.666884+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:02:08.666884+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:19.074054+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":68,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-30T13:02:13.952091+0000","last_change":"2024-03-30T13:01:51.537905+0000","last_active":"2024-03-30T13:02:13.952091+0000","last_peered":"2024-03-30T13:02:13.952091+0000","last_clean":"2024-03-30T13:02:13.952091+0000","last_became_active":"2024-03-30T13:01:41.983624+0000","last_became_peered":"2024-03-30T13:01:41.983624+0000","last_unstale":"2024-03-30T13:02:13.952091+0000","last_undegraded":"2024-03-30T13:02:13.952091+0000","last_fullsized":"2024-03-30T13:02:13.952091+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:01:51.537497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:51.537497+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:55.057623+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":382,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-03-30T13:02:13.227014+0000","last_change":"2024-03-30T13:01:42.036177+0000","last_active":"2024-03-30T13:02:13.227014+0000","last_peered":"2024-03-30T13:02:13.227014+0000","last_clean":"2024-03-30T13:02:13.227014+0000","last_became_active":"2024-03-30T13:01:41.986001+0000","last_became_peered":"2024-03-30T13:01:41.986001+0000","last_unstale":"2024-03-30T13:02:13.227014+0000","last_undegraded":"2024-03-30T13:02:13.227014+0000","last_fullsized":"2024-03-30T13:02:13.227014+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:01:27.958371+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:38.392478+0000","scrub_duration":451,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10453822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":232941852,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106594304,"data_stored":106594304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509525,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48372,"kb_used_data":17196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323468,"statfs":{"total":96636764160,"available":96587231232,"internally_reserved":0,"allocated":17608704,"data_stored":16417738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":156,"seq":670014898187,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54272,"kb_used_data":22580,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94317568,"statfs":{"total":96636764160,"available":96581189632,"internally_reserved":0,"allocated":23121920,"data_stored":21930954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4405,"internal_metadata":32435915},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":135,"seq":579820584982,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45784,"kb_used_data":13644,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94326056,"statfs":{"total":96636764160,"available":96589881344,"internally_reserved":0,"allocated":13971456,"data_stored":12780490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6047,"internal_metadata":32893025},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509526,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45688,"kb_used_data":14508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326152,"statfs":{"total":96636764160,"available":96589979648,"internally_reserved":0,"allocated":14856192,"data_stored":13665226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285611,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42256,"kb_used_data":10564,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94329584,"statfs":{"total":96636764160,"available":96593494016,"internally_reserved":0,"allocated":10817536,"data_stored":9626570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509526,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46768,"kb_used_data":15588,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325072,"statfs":{"total":96636764160,"available":96588873728,"internally_reserved":0,"allocated":15962112,"data_stored":14771146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574934,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41608,"kb_used_data":14588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330232,"statfs":{"total":96636764160,"available":96594157568,"internally_reserved":0,"allocated":14938112,"data_stored":13747146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8679424,"data_stored":8679424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12718080,"data_stored":12718080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20983808,"data_stored":20983808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:02:26.864 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:02:26.864 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:02:26.864 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2024-03-30T13:02:26.864 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-30T13:02:26.864 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-30T13:02:26.867 INFO:tasks.ceph.osd.7:Started 2024-03-30T13:02:26.867 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-30T13:02:26.933 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:02:26.931+0000 7f76257917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:26.939 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:02:26.939+0000 7f76257917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:26.940 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:02:26.939+0000 7f76257917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:26.994 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T13:02:26.994 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T13:02:26.994 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T13:02:26.994 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T13:02:26.994 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T13:02:26.994 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T13:02:26.994 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-03-30T13:02:26.994 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-03-30T13:02:26.994 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-03-30T13:02:26.994 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-03-30T13:02:26.994 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-03-30T13:02:26.994 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T13:02:26.996 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:02:26.996 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-03-30T13:02:27.472 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:02:27.471+0000 7f76257917c0 -1 Falling back to public interface 2024-03-30T13:02:31.251 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:02:31.251+0000 7f76257917c0 -1 osd.7 173 log_to_monitors true 2024-03-30T13:02:31.319 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:02:31.319+0000 7f761955e640 -1 osd.7 173 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:02:31.997 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-30T13:02:32.128 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T13:02:32.128 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T13:02:32.129 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T13:02:32.129 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T13:02:32.142 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:02:32.142 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-03-30T13:02:32.142 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 7 2024-03-30T13:02:32.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:32.268+0000 7f8560b33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:32.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:32.288+0000 7f8560b33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:32.475 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:02:32.472+0000 7fbb38f08640 -1 mon.a@0(leader).osd e180 definitely_dead 0 2024-03-30T13:02:33.316 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.7. 2024-03-30T13:02:33.344 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-03-30T13:02:33.344 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:02:33.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:33.460+0000 7fcb13570640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:33.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:33.476+0000 7fcb13570640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:02:34.344 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:02:34.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:34.460+0000 7fa5a50e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:34.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:34.480+0000 7fa5a50e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:35.326 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:02:35.349 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:02:35.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:35.468+0000 7f736fb89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:35.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:35.480+0000 7f736fb89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:36.338 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:02:36.361 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:02:36.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:36.480+0000 7f48a8e30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:36.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:36.496+0000 7f48a8e30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:37.352 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:02:37.376 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:02:37.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:37.496+0000 7f3ba1860640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:37.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:37.512+0000 7f3ba1860640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:38.365 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:02:38.390 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:02:38.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:38.508+0000 7efc49f30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:38.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:38.524+0000 7efc49f30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:39.368 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:02:39.397 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:02:39.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:39.512+0000 7f5688d89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:39.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:39.528+0000 7f5688d89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:40.392 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:02:40.419 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:02:40.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:40.532+0000 7f10574c2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:40.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:40.548+0000 7f10574c2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:41.413 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:02:41.442 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:02:41.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:41.560+0000 7feafbc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:41.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:41.576+0000 7feafbc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:42.424 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:02:42.452 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:02:42.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:42.568+0000 7f0dced89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:42.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:42.584+0000 7f0dced89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:43.437 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:02:43.460 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:02:43.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:43.576+0000 7f0dcbb98640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:43.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:43.592+0000 7f0dcbb98640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:44.441 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:02:44.470 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:02:44.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:44.588+0000 7f583bb5a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:44.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:44.604+0000 7f583bb5a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:45.457 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:02:45.480 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:02:45.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:45.596+0000 7fc211619640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:45.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:45.612+0000 7fc211619640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:46.470 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:02:46.495 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:02:46.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:46.612+0000 7f91e6e84640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:46.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:46.628+0000 7f91e6e84640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:47.476 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:02:47.500 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:02:47.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:47.616+0000 7f6f923d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:47.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:47.632+0000 7f6f923d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:02:48.515 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:02:48.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:48.636+0000 7f782047c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:48.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:48.652+0000 7f782047c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:49.501 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:02:49.527 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:02:49.527 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:02:49.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:49.644+0000 7f2f1f131640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:49.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:49.660+0000 7f2f1f131640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:49.839 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:02:49.839 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:02:49.855 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":408,"stamp":"2024-03-30T13:02:49.497047+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":240672973,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":360796,"kb_used_data":112100,"kb_used_omap":28,"kb_used_meta":248547,"kb_avail":754613924,"statfs":{"total":773094113280,"available":772724658176,"internally_reserved":0,"allocated":114790400,"data_stored":104513768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28921,"internal_metadata":254512903},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.043927"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":443,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-30T13:02:43.943074+0000","last_change":"2024-03-30T13:01:42.155603+0000","last_active":"2024-03-30T13:02:43.943074+0000","last_peered":"2024-03-30T13:02:43.943074+0000","last_clean":"2024-03-30T13:02:43.943074+0000","last_became_active":"2024-03-30T13:01:41.987359+0000","last_became_peered":"2024-03-30T13:01:41.987359+0000","last_unstale":"2024-03-30T13:02:43.943074+0000","last_undegraded":"2024-03-30T13:02:43.943074+0000","last_fullsized":"2024-03-30T13:02:43.943074+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:01:22.928753+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:22.928753+0000","last_clean_scrub_stamp":"2024-03-30T13:01:22.928753+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:51.214009+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18352980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":403,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-30T13:02:44.446240+0000","last_change":"2024-03-30T13:02:34.363323+0000","last_active":"2024-03-30T13:02:44.446240+0000","last_peered":"2024-03-30T13:02:44.446240+0000","last_clean":"2024-03-30T13:02:44.446240+0000","last_became_active":"2024-03-30T13:02:34.336789+0000","last_became_peered":"2024-03-30T13:02:34.336789+0000","last_unstale":"2024-03-30T13:02:44.446240+0000","last_undegraded":"2024-03-30T13:02:44.446240+0000","last_fullsized":"2024-03-30T13:02:44.446240+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:27.269301+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:02:27.269301+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:36.631791+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":38,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-30T13:02:44.445401+0000","last_change":"2024-03-30T13:02:34.415429+0000","last_active":"2024-03-30T13:02:44.445401+0000","last_peered":"2024-03-30T13:02:44.445401+0000","last_clean":"2024-03-30T13:02:44.445401+0000","last_became_active":"2024-03-30T13:02:34.331788+0000","last_became_peered":"2024-03-30T13:02:34.331788+0000","last_unstale":"2024-03-30T13:02:44.445401+0000","last_undegraded":"2024-03-30T13:02:44.445401+0000","last_fullsized":"2024-03-30T13:02:44.445401+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:02:33.302956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:33.302956+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:35.749650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":36,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-30T13:02:44.445256+0000","last_change":"2024-03-30T13:02:34.996784+0000","last_active":"2024-03-30T13:02:44.445256+0000","last_peered":"2024-03-30T13:02:44.445256+0000","last_clean":"2024-03-30T13:02:44.445256+0000","last_became_active":"2024-03-30T13:02:34.963470+0000","last_became_peered":"2024-03-30T13:02:34.963470+0000","last_unstale":"2024-03-30T13:02:44.445256+0000","last_undegraded":"2024-03-30T13:02:44.445256+0000","last_fullsized":"2024-03-30T13:02:44.445256+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:02:02.359931+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:02:02.359931+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:12.493904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":89,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-30T13:02:45.164820+0000","last_change":"2024-03-30T13:02:35.984755+0000","last_active":"2024-03-30T13:02:45.164820+0000","last_peered":"2024-03-30T13:02:45.164820+0000","last_clean":"2024-03-30T13:02:45.164820+0000","last_became_active":"2024-03-30T13:01:41.984313+0000","last_became_peered":"2024-03-30T13:01:41.984313+0000","last_unstale":"2024-03-30T13:02:45.164820+0000","last_undegraded":"2024-03-30T13:02:45.164820+0000","last_fullsized":"2024-03-30T13:02:45.164820+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:35.984506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:35.984506+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:55.717882+0000","scrub_duration":291,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":448,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-03-30T13:02:47.458812+0000","last_change":"2024-03-30T13:02:10.596835+0000","last_active":"2024-03-30T13:02:47.458812+0000","last_peered":"2024-03-30T13:02:47.458812+0000","last_clean":"2024-03-30T13:02:47.458812+0000","last_became_active":"2024-03-30T13:01:41.987920+0000","last_became_peered":"2024-03-30T13:01:41.987920+0000","last_unstale":"2024-03-30T13:02:47.458812+0000","last_undegraded":"2024-03-30T13:02:47.458812+0000","last_fullsized":"2024-03-30T13:02:47.458812+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:02:10.596458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:10.596458+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:38.665640+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22789962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":51,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-30T13:02:44.445345+0000","last_change":"2024-03-30T13:02:34.570436+0000","last_active":"2024-03-30T13:02:44.445345+0000","last_peered":"2024-03-30T13:02:44.445345+0000","last_clean":"2024-03-30T13:02:44.445345+0000","last_became_active":"2024-03-30T13:02:34.332732+0000","last_became_peered":"2024-03-30T13:02:34.332732+0000","last_unstale":"2024-03-30T13:02:44.445345+0000","last_undegraded":"2024-03-30T13:02:44.445345+0000","last_fullsized":"2024-03-30T13:02:44.445345+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:01:48.791509+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:01:48.791509+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:54.580526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":436,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-03-30T13:02:46.468050+0000","last_change":"2024-03-30T13:02:37.318261+0000","last_active":"2024-03-30T13:02:46.468050+0000","last_peered":"2024-03-30T13:02:46.468050+0000","last_clean":"2024-03-30T13:02:46.468050+0000","last_became_active":"2024-03-30T13:01:41.988151+0000","last_became_peered":"2024-03-30T13:01:41.988151+0000","last_unstale":"2024-03-30T13:02:46.468050+0000","last_undegraded":"2024-03-30T13:02:46.468050+0000","last_fullsized":"2024-03-30T13:02:46.468050+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:02:37.317994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:37.317994+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:02.570664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":22033680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":500,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-03-30T13:02:47.459234+0000","last_change":"2024-03-30T13:02:34.376265+0000","last_active":"2024-03-30T13:02:47.459234+0000","last_peered":"2024-03-30T13:02:47.459234+0000","last_clean":"2024-03-30T13:02:47.459234+0000","last_became_active":"2024-03-30T13:02:34.338628+0000","last_became_peered":"2024-03-30T13:02:34.338628+0000","last_unstale":"2024-03-30T13:02:47.459234+0000","last_undegraded":"2024-03-30T13:02:47.459234+0000","last_fullsized":"2024-03-30T13:02:47.459234+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:02:08.597557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:08.597557+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:23.617852+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":40605184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":320,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-30T13:02:44.446076+0000","last_change":"2024-03-30T13:02:28.264089+0000","last_active":"2024-03-30T13:02:44.446076+0000","last_peered":"2024-03-30T13:02:44.446076+0000","last_clean":"2024-03-30T13:02:44.446076+0000","last_became_active":"2024-03-30T13:00:46.109054+0000","last_became_peered":"2024-03-30T13:00:46.109054+0000","last_unstale":"2024-03-30T13:02:44.446076+0000","last_undegraded":"2024-03-30T13:02:44.446076+0000","last_fullsized":"2024-03-30T13:02:44.446076+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:28.263717+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:02:28.263717+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:56.652431+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":455,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-30T13:02:43.943037+0000","last_change":"2024-03-30T13:02:18.934000+0000","last_active":"2024-03-30T13:02:43.943037+0000","last_peered":"2024-03-30T13:02:43.943037+0000","last_clean":"2024-03-30T13:02:43.943037+0000","last_became_active":"2024-03-30T13:00:46.106211+0000","last_became_peered":"2024-03-30T13:00:46.106211+0000","last_unstale":"2024-03-30T13:02:43.943037+0000","last_undegraded":"2024-03-30T13:02:43.943037+0000","last_fullsized":"2024-03-30T13:02:43.943037+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:02:18.933526+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:42.090625+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25479082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":463,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-30T13:02:43.943180+0000","last_change":"2024-03-30T13:02:24.920510+0000","last_active":"2024-03-30T13:02:43.943180+0000","last_peered":"2024-03-30T13:02:43.943180+0000","last_clean":"2024-03-30T13:02:43.943180+0000","last_became_active":"2024-03-30T13:01:41.991185+0000","last_became_peered":"2024-03-30T13:01:41.991185+0000","last_unstale":"2024-03-30T13:02:43.943180+0000","last_undegraded":"2024-03-30T13:02:43.943180+0000","last_fullsized":"2024-03-30T13:02:43.943180+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:02:24.920236+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T13:02:24.920236+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:28.373546+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":95,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-30T13:02:45.164820+0000","last_change":"2024-03-30T13:01:53.533883+0000","last_active":"2024-03-30T13:02:45.164820+0000","last_peered":"2024-03-30T13:02:45.164820+0000","last_clean":"2024-03-30T13:02:45.164820+0000","last_became_active":"2024-03-30T13:01:41.982566+0000","last_became_peered":"2024-03-30T13:01:41.982566+0000","last_unstale":"2024-03-30T13:02:45.164820+0000","last_undegraded":"2024-03-30T13:02:45.164820+0000","last_fullsized":"2024-03-30T13:02:45.164820+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:01:53.533647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:53.533647+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:00.040107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":34,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-30T13:02:44.445304+0000","last_change":"2024-03-30T13:02:34.360717+0000","last_active":"2024-03-30T13:02:44.445304+0000","last_peered":"2024-03-30T13:02:44.445304+0000","last_clean":"2024-03-30T13:02:44.445304+0000","last_became_active":"2024-03-30T13:02:34.331686+0000","last_became_peered":"2024-03-30T13:02:34.331686+0000","last_unstale":"2024-03-30T13:02:44.445304+0000","last_undegraded":"2024-03-30T13:02:44.445304+0000","last_fullsized":"2024-03-30T13:02:44.445304+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:08.666884+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:02:08.666884+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:11.626036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":95,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-30T13:02:45.164855+0000","last_change":"2024-03-30T13:01:51.537905+0000","last_active":"2024-03-30T13:02:45.164855+0000","last_peered":"2024-03-30T13:02:45.164855+0000","last_clean":"2024-03-30T13:02:45.164855+0000","last_became_active":"2024-03-30T13:01:41.983624+0000","last_became_peered":"2024-03-30T13:01:41.983624+0000","last_unstale":"2024-03-30T13:02:45.164855+0000","last_undegraded":"2024-03-30T13:02:45.164855+0000","last_fullsized":"2024-03-30T13:02:45.164855+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:01:51.537497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:51.537497+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:55.057623+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":422,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-03-30T13:02:46.468168+0000","last_change":"2024-03-30T13:02:40.333877+0000","last_active":"2024-03-30T13:02:46.468168+0000","last_peered":"2024-03-30T13:02:46.468168+0000","last_clean":"2024-03-30T13:02:46.468168+0000","last_became_active":"2024-03-30T13:01:41.986001+0000","last_became_peered":"2024-03-30T13:01:41.986001+0000","last_unstale":"2024-03-30T13:02:46.468168+0000","last_undegraded":"2024-03-30T13:02:46.468168+0000","last_fullsized":"2024-03-30T13:02:46.468168+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:02:40.333589+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:02:40.333589+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:45.470373+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10453822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":240672973,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509529,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48536,"kb_used_data":17352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323304,"statfs":{"total":96636764160,"available":96587063296,"internally_reserved":0,"allocated":17768448,"data_stored":16490670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":156,"seq":670014898192,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53800,"kb_used_data":22108,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94318040,"statfs":{"total":96636764160,"available":96581672960,"internally_reserved":0,"allocated":22638592,"data_stored":21354013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4405,"internal_metadata":32435915},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":135,"seq":579820584986,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45952,"kb_used_data":13800,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94325888,"statfs":{"total":96636764160,"available":96589709312,"internally_reserved":0,"allocated":14131200,"data_stored":12853422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6047,"internal_metadata":32893025},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509530,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39748,"kb_used_data":8568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332092,"statfs":{"total":96636764160,"available":96596062208,"internally_reserved":0,"allocated":8773632,"data_stored":7495854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285616,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39144,"kb_used_data":7448,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94332696,"statfs":{"total":96636764160,"available":96596680704,"internally_reserved":0,"allocated":7626752,"data_stored":6328571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509530,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46924,"kb_used_data":15744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324916,"statfs":{"total":96636764160,"available":96588713984,"internally_reserved":0,"allocated":16121856,"data_stored":14844078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":180,"seq":773094113285,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44892,"kb_used_data":12300,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94326948,"statfs":{"total":96636764160,"available":96590794752,"internally_reserved":0,"allocated":12595200,"data_stored":11310621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7690,"internal_metadata":33350134},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574939,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41800,"kb_used_data":14780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330040,"statfs":{"total":96636764160,"available":96593960960,"internally_reserved":0,"allocated":15134720,"data_stored":13836539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:02:49.856 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:02:49.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:49.964+0000 7f5a17daf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:49.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:49.980+0000 7f5a17daf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:50.202 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:02:50.203 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:02:50.223 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-03-30T13:02:49.907262+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":240672973,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":360992,"kb_used_data":112280,"kb_used_omap":28,"kb_used_meta":248547,"kb_avail":754613728,"statfs":{"total":773094113280,"available":772724457472,"internally_reserved":0,"allocated":114974720,"data_stored":104596073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28921,"internal_metadata":254512903},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001085"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":451,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:02:48.544439+0000","last_change":"2024-03-30T13:01:42.155603+0000","last_active":"2024-03-30T13:02:48.544439+0000","last_peered":"2024-03-30T13:02:48.544439+0000","last_clean":"2024-03-30T13:02:48.544439+0000","last_became_active":"2024-03-30T13:01:41.987359+0000","last_became_peered":"2024-03-30T13:01:41.987359+0000","last_unstale":"2024-03-30T13:02:48.544439+0000","last_undegraded":"2024-03-30T13:02:48.544439+0000","last_fullsized":"2024-03-30T13:02:48.544439+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:01:22.928753+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:22.928753+0000","last_clean_scrub_stamp":"2024-03-30T13:01:22.928753+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:51.214009+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18352980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":403,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-30T13:02:44.446240+0000","last_change":"2024-03-30T13:02:34.363323+0000","last_active":"2024-03-30T13:02:44.446240+0000","last_peered":"2024-03-30T13:02:44.446240+0000","last_clean":"2024-03-30T13:02:44.446240+0000","last_became_active":"2024-03-30T13:02:34.336789+0000","last_became_peered":"2024-03-30T13:02:34.336789+0000","last_unstale":"2024-03-30T13:02:44.446240+0000","last_undegraded":"2024-03-30T13:02:44.446240+0000","last_fullsized":"2024-03-30T13:02:44.446240+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:27.269301+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:02:27.269301+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:36.631791+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":38,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-30T13:02:44.445401+0000","last_change":"2024-03-30T13:02:34.415429+0000","last_active":"2024-03-30T13:02:44.445401+0000","last_peered":"2024-03-30T13:02:44.445401+0000","last_clean":"2024-03-30T13:02:44.445401+0000","last_became_active":"2024-03-30T13:02:34.331788+0000","last_became_peered":"2024-03-30T13:02:34.331788+0000","last_unstale":"2024-03-30T13:02:44.445401+0000","last_undegraded":"2024-03-30T13:02:44.445401+0000","last_fullsized":"2024-03-30T13:02:44.445401+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:02:33.302956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:33.302956+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:35.749650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":36,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-30T13:02:44.445256+0000","last_change":"2024-03-30T13:02:34.996784+0000","last_active":"2024-03-30T13:02:44.445256+0000","last_peered":"2024-03-30T13:02:44.445256+0000","last_clean":"2024-03-30T13:02:44.445256+0000","last_became_active":"2024-03-30T13:02:34.963470+0000","last_became_peered":"2024-03-30T13:02:34.963470+0000","last_unstale":"2024-03-30T13:02:44.445256+0000","last_undegraded":"2024-03-30T13:02:44.445256+0000","last_fullsized":"2024-03-30T13:02:44.445256+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:02:02.359931+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:02:02.359931+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:12.493904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":89,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-30T13:02:45.164820+0000","last_change":"2024-03-30T13:02:35.984755+0000","last_active":"2024-03-30T13:02:45.164820+0000","last_peered":"2024-03-30T13:02:45.164820+0000","last_clean":"2024-03-30T13:02:45.164820+0000","last_became_active":"2024-03-30T13:01:41.984313+0000","last_became_peered":"2024-03-30T13:01:41.984313+0000","last_unstale":"2024-03-30T13:02:45.164820+0000","last_undegraded":"2024-03-30T13:02:45.164820+0000","last_fullsized":"2024-03-30T13:02:45.164820+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:35.984506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:35.984506+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:55.717882+0000","scrub_duration":291,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":448,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-03-30T13:02:47.458812+0000","last_change":"2024-03-30T13:02:10.596835+0000","last_active":"2024-03-30T13:02:47.458812+0000","last_peered":"2024-03-30T13:02:47.458812+0000","last_clean":"2024-03-30T13:02:47.458812+0000","last_became_active":"2024-03-30T13:01:41.987920+0000","last_became_peered":"2024-03-30T13:01:41.987920+0000","last_unstale":"2024-03-30T13:02:47.458812+0000","last_undegraded":"2024-03-30T13:02:47.458812+0000","last_fullsized":"2024-03-30T13:02:47.458812+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:02:10.596458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:10.596458+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:38.665640+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22789962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":51,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-30T13:02:44.445345+0000","last_change":"2024-03-30T13:02:34.570436+0000","last_active":"2024-03-30T13:02:44.445345+0000","last_peered":"2024-03-30T13:02:44.445345+0000","last_clean":"2024-03-30T13:02:44.445345+0000","last_became_active":"2024-03-30T13:02:34.332732+0000","last_became_peered":"2024-03-30T13:02:34.332732+0000","last_unstale":"2024-03-30T13:02:44.445345+0000","last_undegraded":"2024-03-30T13:02:44.445345+0000","last_fullsized":"2024-03-30T13:02:44.445345+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:01:48.791509+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:01:48.791509+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:54.580526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":436,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-03-30T13:02:46.468050+0000","last_change":"2024-03-30T13:02:37.318261+0000","last_active":"2024-03-30T13:02:46.468050+0000","last_peered":"2024-03-30T13:02:46.468050+0000","last_clean":"2024-03-30T13:02:46.468050+0000","last_became_active":"2024-03-30T13:01:41.988151+0000","last_became_peered":"2024-03-30T13:01:41.988151+0000","last_unstale":"2024-03-30T13:02:46.468050+0000","last_undegraded":"2024-03-30T13:02:46.468050+0000","last_fullsized":"2024-03-30T13:02:46.468050+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:02:37.317994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:37.317994+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:02.570664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":22033680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":500,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-03-30T13:02:47.459234+0000","last_change":"2024-03-30T13:02:34.376265+0000","last_active":"2024-03-30T13:02:47.459234+0000","last_peered":"2024-03-30T13:02:47.459234+0000","last_clean":"2024-03-30T13:02:47.459234+0000","last_became_active":"2024-03-30T13:02:34.338628+0000","last_became_peered":"2024-03-30T13:02:34.338628+0000","last_unstale":"2024-03-30T13:02:47.459234+0000","last_undegraded":"2024-03-30T13:02:47.459234+0000","last_fullsized":"2024-03-30T13:02:47.459234+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:02:08.597557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:08.597557+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:23.617852+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":40605184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":320,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-30T13:02:44.446076+0000","last_change":"2024-03-30T13:02:28.264089+0000","last_active":"2024-03-30T13:02:44.446076+0000","last_peered":"2024-03-30T13:02:44.446076+0000","last_clean":"2024-03-30T13:02:44.446076+0000","last_became_active":"2024-03-30T13:00:46.109054+0000","last_became_peered":"2024-03-30T13:00:46.109054+0000","last_unstale":"2024-03-30T13:02:44.446076+0000","last_undegraded":"2024-03-30T13:02:44.446076+0000","last_fullsized":"2024-03-30T13:02:44.446076+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:28.263717+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:02:28.263717+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:56.652431+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":463,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:02:48.544397+0000","last_change":"2024-03-30T13:02:18.934000+0000","last_active":"2024-03-30T13:02:48.544397+0000","last_peered":"2024-03-30T13:02:48.544397+0000","last_clean":"2024-03-30T13:02:48.544397+0000","last_became_active":"2024-03-30T13:00:46.106211+0000","last_became_peered":"2024-03-30T13:00:46.106211+0000","last_unstale":"2024-03-30T13:02:48.544397+0000","last_undegraded":"2024-03-30T13:02:48.544397+0000","last_fullsized":"2024-03-30T13:02:48.544397+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:02:18.933526+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:42.090625+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25479082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":471,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:02:48.544659+0000","last_change":"2024-03-30T13:02:24.920510+0000","last_active":"2024-03-30T13:02:48.544659+0000","last_peered":"2024-03-30T13:02:48.544659+0000","last_clean":"2024-03-30T13:02:48.544659+0000","last_became_active":"2024-03-30T13:01:41.991185+0000","last_became_peered":"2024-03-30T13:01:41.991185+0000","last_unstale":"2024-03-30T13:02:48.544659+0000","last_undegraded":"2024-03-30T13:02:48.544659+0000","last_fullsized":"2024-03-30T13:02:48.544659+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:02:24.920236+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T13:02:24.920236+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:28.373546+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":95,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-30T13:02:45.164820+0000","last_change":"2024-03-30T13:01:53.533883+0000","last_active":"2024-03-30T13:02:45.164820+0000","last_peered":"2024-03-30T13:02:45.164820+0000","last_clean":"2024-03-30T13:02:45.164820+0000","last_became_active":"2024-03-30T13:01:41.982566+0000","last_became_peered":"2024-03-30T13:01:41.982566+0000","last_unstale":"2024-03-30T13:02:45.164820+0000","last_undegraded":"2024-03-30T13:02:45.164820+0000","last_fullsized":"2024-03-30T13:02:45.164820+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:01:53.533647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:53.533647+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:00.040107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":34,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-30T13:02:44.445304+0000","last_change":"2024-03-30T13:02:34.360717+0000","last_active":"2024-03-30T13:02:44.445304+0000","last_peered":"2024-03-30T13:02:44.445304+0000","last_clean":"2024-03-30T13:02:44.445304+0000","last_became_active":"2024-03-30T13:02:34.331686+0000","last_became_peered":"2024-03-30T13:02:34.331686+0000","last_unstale":"2024-03-30T13:02:44.445304+0000","last_undegraded":"2024-03-30T13:02:44.445304+0000","last_fullsized":"2024-03-30T13:02:44.445304+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:08.666884+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:02:08.666884+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:11.626036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":95,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-30T13:02:45.164855+0000","last_change":"2024-03-30T13:01:51.537905+0000","last_active":"2024-03-30T13:02:45.164855+0000","last_peered":"2024-03-30T13:02:45.164855+0000","last_clean":"2024-03-30T13:02:45.164855+0000","last_became_active":"2024-03-30T13:01:41.983624+0000","last_became_peered":"2024-03-30T13:01:41.983624+0000","last_unstale":"2024-03-30T13:02:45.164855+0000","last_undegraded":"2024-03-30T13:02:45.164855+0000","last_fullsized":"2024-03-30T13:02:45.164855+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:01:51.537497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:51.537497+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:55.057623+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":422,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-03-30T13:02:46.468168+0000","last_change":"2024-03-30T13:02:40.333877+0000","last_active":"2024-03-30T13:02:46.468168+0000","last_peered":"2024-03-30T13:02:46.468168+0000","last_clean":"2024-03-30T13:02:46.468168+0000","last_became_active":"2024-03-30T13:01:41.986001+0000","last_became_peered":"2024-03-30T13:01:41.986001+0000","last_unstale":"2024-03-30T13:02:46.468168+0000","last_undegraded":"2024-03-30T13:02:46.468168+0000","last_fullsized":"2024-03-30T13:02:46.468168+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:02:40.333589+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:02:40.333589+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:45.470373+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10453822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":240672973,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509530,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48592,"kb_used_data":17412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323248,"statfs":{"total":96636764160,"available":96587005952,"internally_reserved":0,"allocated":17829888,"data_stored":16518105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":156,"seq":670014898192,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53800,"kb_used_data":22108,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94318040,"statfs":{"total":96636764160,"available":96581672960,"internally_reserved":0,"allocated":22638592,"data_stored":21354013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4405,"internal_metadata":32435915},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":135,"seq":579820584986,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45952,"kb_used_data":13800,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94325888,"statfs":{"total":96636764160,"available":96589709312,"internally_reserved":0,"allocated":14131200,"data_stored":12853422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6047,"internal_metadata":32893025},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509531,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39820,"kb_used_data":8628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332020,"statfs":{"total":96636764160,"available":96595988480,"internally_reserved":0,"allocated":8835072,"data_stored":7523289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285616,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39144,"kb_used_data":7448,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94332696,"statfs":{"total":96636764160,"available":96596680704,"internally_reserved":0,"allocated":7626752,"data_stored":6328571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509531,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46992,"kb_used_data":15804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324848,"statfs":{"total":96636764160,"available":96588644352,"internally_reserved":0,"allocated":16183296,"data_stored":14871513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":180,"seq":773094113285,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44892,"kb_used_data":12300,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94326948,"statfs":{"total":96636764160,"available":96590794752,"internally_reserved":0,"allocated":12595200,"data_stored":11310621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7690,"internal_metadata":33350134},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574939,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41800,"kb_used_data":14780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330040,"statfs":{"total":96636764160,"available":96593960960,"internally_reserved":0,"allocated":15134720,"data_stored":13836539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:02:50.224 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:02:50.224 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:02:50.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:50.340+0000 7f71e6c9b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:50.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:50.356+0000 7f71e6c9b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:50.534 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:02:50.535 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":197,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:02:49.493702+0000","last_up_change":"2024-03-30T13:02:32.283721+0000","last_in_change":"2024-03-30T13:02:33.293676+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":13,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":77,"up_from":83,"up_thru":175,"down_at":78,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6818","nonce":3104224893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6821","nonce":3104224893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6828","nonce":3104224893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6825","nonce":3104224893}]},"public_addr":"172.21.15.67:6818/3104224893","cluster_addr":"172.21.15.67:6821/3104224893","heartbeat_back_addr":"172.21.15.67:6828/3104224893","heartbeat_front_addr":"172.21.15.67:6825/3104224893","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":175,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":181,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6801","nonce":1504348724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6803","nonce":1504348724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6807","nonce":1504348724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6805","nonce":1504348724}]},"public_addr":"172.21.15.67:6801/1504348724","cluster_addr":"172.21.15.67:6803/1504348724","heartbeat_back_addr":"172.21.15.67:6807/1504348724","heartbeat_front_addr":"172.21.15.67:6805/1504348724","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":36,"last_clean_end":125,"up_from":135,"up_thru":181,"down_at":126,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":2818291404},{"type":"v1","addr":"172.21.15.82:6803","nonce":2818291404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":2818291404},{"type":"v1","addr":"172.21.15.82:6807","nonce":2818291404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":2818291404},{"type":"v1","addr":"172.21.15.82:6815","nonce":2818291404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":2818291404},{"type":"v1","addr":"172.21.15.82:6811","nonce":2818291404}]},"public_addr":"172.21.15.82:6803/2818291404","cluster_addr":"172.21.15.82:6807/2818291404","heartbeat_back_addr":"172.21.15.82:6815/2818291404","heartbeat_front_addr":"172.21.15.82:6811/2818291404","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":151,"up_from":156,"up_thru":175,"down_at":152,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":3428934162},{"type":"v1","addr":"172.21.15.82:6817","nonce":3428934162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":3428934162},{"type":"v1","addr":"172.21.15.82:6819","nonce":3428934162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":3428934162},{"type":"v1","addr":"172.21.15.82:6823","nonce":3428934162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":3428934162},{"type":"v1","addr":"172.21.15.82:6821","nonce":3428934162}]},"public_addr":"172.21.15.82:6817/3428934162","cluster_addr":"172.21.15.82:6819/3428934162","heartbeat_back_addr":"172.21.15.82:6823/3428934162","heartbeat_front_addr":"172.21.15.82:6821/3428934162","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6801","nonce":968508548}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6806","nonce":968508548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6814","nonce":968508548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6810","nonce":968508548}]},"public_addr":"172.21.15.82:6801/968508548","cluster_addr":"172.21.15.82:6806/968508548","heartbeat_back_addr":"172.21.15.82:6814/968508548","heartbeat_front_addr":"172.21.15.82:6810/968508548","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":107,"last_clean_end":173,"up_from":180,"up_thru":181,"down_at":174,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2052802708},{"type":"v1","addr":"172.21.15.82:6825","nonce":2052802708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2052802708},{"type":"v1","addr":"172.21.15.82:6827","nonce":2052802708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2052802708},{"type":"v1","addr":"172.21.15.82:6831","nonce":2052802708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2052802708},{"type":"v1","addr":"172.21.15.82:6829","nonce":2052802708}]},"public_addr":"172.21.15.82:6825/2052802708","cluster_addr":"172.21.15.82:6827/2052802708","heartbeat_back_addr":"172.21.15.82:6831/2052802708","heartbeat_front_addr":"172.21.15.82:6829/2052802708","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:01:45.024311+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:01:45.724887+0000","dead_epoch":78},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:01:44.194796+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:02:02.536853+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:02:12.610067+0000","dead_epoch":126},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:01:41.431935+0000","dead_epoch":152},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:01:40.144896+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:02:32.251172+0000","dead_epoch":174}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:02:50.553 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:02:50.554 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:02:50.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:50.656+0000 7fcf92abd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:50.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:50.672+0000 7fcf92abd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:50.837 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:02:50.854 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:02:50.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:50.956+0000 7fe026490640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:50.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:50.972+0000 7fe026490640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:51.156 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:02:51.156 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:02:51.175 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:02:51.175 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:02:51.175 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:02:51.175 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:02:51.175 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 3, 6, 1, 4, 5, 7] 2024-03-30T13:02:51.175 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:02:51.175 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:02:51.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:51.280+0000 7fb9ba544640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:51.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:51.292+0000 7fb9ba544640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:51.468 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:02:51.468 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:02:51.488 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-03-30T13:02:49.907262+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":240672973,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":360992,"kb_used_data":112280,"kb_used_omap":28,"kb_used_meta":248547,"kb_avail":754613728,"statfs":{"total":773094113280,"available":772724457472,"internally_reserved":0,"allocated":114974720,"data_stored":104596073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28921,"internal_metadata":254512903},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001085"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":451,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:02:48.544439+0000","last_change":"2024-03-30T13:01:42.155603+0000","last_active":"2024-03-30T13:02:48.544439+0000","last_peered":"2024-03-30T13:02:48.544439+0000","last_clean":"2024-03-30T13:02:48.544439+0000","last_became_active":"2024-03-30T13:01:41.987359+0000","last_became_peered":"2024-03-30T13:01:41.987359+0000","last_unstale":"2024-03-30T13:02:48.544439+0000","last_undegraded":"2024-03-30T13:02:48.544439+0000","last_fullsized":"2024-03-30T13:02:48.544439+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:01:22.928753+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:22.928753+0000","last_clean_scrub_stamp":"2024-03-30T13:01:22.928753+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:51.214009+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18352980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":403,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-30T13:02:44.446240+0000","last_change":"2024-03-30T13:02:34.363323+0000","last_active":"2024-03-30T13:02:44.446240+0000","last_peered":"2024-03-30T13:02:44.446240+0000","last_clean":"2024-03-30T13:02:44.446240+0000","last_became_active":"2024-03-30T13:02:34.336789+0000","last_became_peered":"2024-03-30T13:02:34.336789+0000","last_unstale":"2024-03-30T13:02:44.446240+0000","last_undegraded":"2024-03-30T13:02:44.446240+0000","last_fullsized":"2024-03-30T13:02:44.446240+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:27.269301+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:02:27.269301+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:36.631791+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":38,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-30T13:02:44.445401+0000","last_change":"2024-03-30T13:02:34.415429+0000","last_active":"2024-03-30T13:02:44.445401+0000","last_peered":"2024-03-30T13:02:44.445401+0000","last_clean":"2024-03-30T13:02:44.445401+0000","last_became_active":"2024-03-30T13:02:34.331788+0000","last_became_peered":"2024-03-30T13:02:34.331788+0000","last_unstale":"2024-03-30T13:02:44.445401+0000","last_undegraded":"2024-03-30T13:02:44.445401+0000","last_fullsized":"2024-03-30T13:02:44.445401+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:02:33.302956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:33.302956+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:35.749650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":36,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-30T13:02:44.445256+0000","last_change":"2024-03-30T13:02:34.996784+0000","last_active":"2024-03-30T13:02:44.445256+0000","last_peered":"2024-03-30T13:02:44.445256+0000","last_clean":"2024-03-30T13:02:44.445256+0000","last_became_active":"2024-03-30T13:02:34.963470+0000","last_became_peered":"2024-03-30T13:02:34.963470+0000","last_unstale":"2024-03-30T13:02:44.445256+0000","last_undegraded":"2024-03-30T13:02:44.445256+0000","last_fullsized":"2024-03-30T13:02:44.445256+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:02:02.359931+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:02:02.359931+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:12.493904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":89,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-30T13:02:45.164820+0000","last_change":"2024-03-30T13:02:35.984755+0000","last_active":"2024-03-30T13:02:45.164820+0000","last_peered":"2024-03-30T13:02:45.164820+0000","last_clean":"2024-03-30T13:02:45.164820+0000","last_became_active":"2024-03-30T13:01:41.984313+0000","last_became_peered":"2024-03-30T13:01:41.984313+0000","last_unstale":"2024-03-30T13:02:45.164820+0000","last_undegraded":"2024-03-30T13:02:45.164820+0000","last_fullsized":"2024-03-30T13:02:45.164820+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:35.984506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:35.984506+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:55.717882+0000","scrub_duration":291,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":448,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-03-30T13:02:47.458812+0000","last_change":"2024-03-30T13:02:10.596835+0000","last_active":"2024-03-30T13:02:47.458812+0000","last_peered":"2024-03-30T13:02:47.458812+0000","last_clean":"2024-03-30T13:02:47.458812+0000","last_became_active":"2024-03-30T13:01:41.987920+0000","last_became_peered":"2024-03-30T13:01:41.987920+0000","last_unstale":"2024-03-30T13:02:47.458812+0000","last_undegraded":"2024-03-30T13:02:47.458812+0000","last_fullsized":"2024-03-30T13:02:47.458812+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:02:10.596458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:10.596458+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:38.665640+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22789962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":51,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-30T13:02:44.445345+0000","last_change":"2024-03-30T13:02:34.570436+0000","last_active":"2024-03-30T13:02:44.445345+0000","last_peered":"2024-03-30T13:02:44.445345+0000","last_clean":"2024-03-30T13:02:44.445345+0000","last_became_active":"2024-03-30T13:02:34.332732+0000","last_became_peered":"2024-03-30T13:02:34.332732+0000","last_unstale":"2024-03-30T13:02:44.445345+0000","last_undegraded":"2024-03-30T13:02:44.445345+0000","last_fullsized":"2024-03-30T13:02:44.445345+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:01:48.791509+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:01:48.791509+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:54.580526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":436,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-03-30T13:02:46.468050+0000","last_change":"2024-03-30T13:02:37.318261+0000","last_active":"2024-03-30T13:02:46.468050+0000","last_peered":"2024-03-30T13:02:46.468050+0000","last_clean":"2024-03-30T13:02:46.468050+0000","last_became_active":"2024-03-30T13:01:41.988151+0000","last_became_peered":"2024-03-30T13:01:41.988151+0000","last_unstale":"2024-03-30T13:02:46.468050+0000","last_undegraded":"2024-03-30T13:02:46.468050+0000","last_fullsized":"2024-03-30T13:02:46.468050+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:02:37.317994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:37.317994+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:02.570664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":22033680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":500,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-03-30T13:02:47.459234+0000","last_change":"2024-03-30T13:02:34.376265+0000","last_active":"2024-03-30T13:02:47.459234+0000","last_peered":"2024-03-30T13:02:47.459234+0000","last_clean":"2024-03-30T13:02:47.459234+0000","last_became_active":"2024-03-30T13:02:34.338628+0000","last_became_peered":"2024-03-30T13:02:34.338628+0000","last_unstale":"2024-03-30T13:02:47.459234+0000","last_undegraded":"2024-03-30T13:02:47.459234+0000","last_fullsized":"2024-03-30T13:02:47.459234+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:02:08.597557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:08.597557+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:23.617852+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":40605184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":320,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-30T13:02:44.446076+0000","last_change":"2024-03-30T13:02:28.264089+0000","last_active":"2024-03-30T13:02:44.446076+0000","last_peered":"2024-03-30T13:02:44.446076+0000","last_clean":"2024-03-30T13:02:44.446076+0000","last_became_active":"2024-03-30T13:00:46.109054+0000","last_became_peered":"2024-03-30T13:00:46.109054+0000","last_unstale":"2024-03-30T13:02:44.446076+0000","last_undegraded":"2024-03-30T13:02:44.446076+0000","last_fullsized":"2024-03-30T13:02:44.446076+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:28.263717+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:02:28.263717+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:56.652431+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":463,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:02:48.544397+0000","last_change":"2024-03-30T13:02:18.934000+0000","last_active":"2024-03-30T13:02:48.544397+0000","last_peered":"2024-03-30T13:02:48.544397+0000","last_clean":"2024-03-30T13:02:48.544397+0000","last_became_active":"2024-03-30T13:00:46.106211+0000","last_became_peered":"2024-03-30T13:00:46.106211+0000","last_unstale":"2024-03-30T13:02:48.544397+0000","last_undegraded":"2024-03-30T13:02:48.544397+0000","last_fullsized":"2024-03-30T13:02:48.544397+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:02:18.933526+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:42.090625+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25479082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":471,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:02:48.544659+0000","last_change":"2024-03-30T13:02:24.920510+0000","last_active":"2024-03-30T13:02:48.544659+0000","last_peered":"2024-03-30T13:02:48.544659+0000","last_clean":"2024-03-30T13:02:48.544659+0000","last_became_active":"2024-03-30T13:01:41.991185+0000","last_became_peered":"2024-03-30T13:01:41.991185+0000","last_unstale":"2024-03-30T13:02:48.544659+0000","last_undegraded":"2024-03-30T13:02:48.544659+0000","last_fullsized":"2024-03-30T13:02:48.544659+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:02:24.920236+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T13:02:24.920236+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:28.373546+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":95,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-30T13:02:45.164820+0000","last_change":"2024-03-30T13:01:53.533883+0000","last_active":"2024-03-30T13:02:45.164820+0000","last_peered":"2024-03-30T13:02:45.164820+0000","last_clean":"2024-03-30T13:02:45.164820+0000","last_became_active":"2024-03-30T13:01:41.982566+0000","last_became_peered":"2024-03-30T13:01:41.982566+0000","last_unstale":"2024-03-30T13:02:45.164820+0000","last_undegraded":"2024-03-30T13:02:45.164820+0000","last_fullsized":"2024-03-30T13:02:45.164820+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:01:53.533647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:53.533647+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:00.040107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":34,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-30T13:02:44.445304+0000","last_change":"2024-03-30T13:02:34.360717+0000","last_active":"2024-03-30T13:02:44.445304+0000","last_peered":"2024-03-30T13:02:44.445304+0000","last_clean":"2024-03-30T13:02:44.445304+0000","last_became_active":"2024-03-30T13:02:34.331686+0000","last_became_peered":"2024-03-30T13:02:34.331686+0000","last_unstale":"2024-03-30T13:02:44.445304+0000","last_undegraded":"2024-03-30T13:02:44.445304+0000","last_fullsized":"2024-03-30T13:02:44.445304+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:08.666884+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:02:08.666884+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:11.626036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":95,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-30T13:02:45.164855+0000","last_change":"2024-03-30T13:01:51.537905+0000","last_active":"2024-03-30T13:02:45.164855+0000","last_peered":"2024-03-30T13:02:45.164855+0000","last_clean":"2024-03-30T13:02:45.164855+0000","last_became_active":"2024-03-30T13:01:41.983624+0000","last_became_peered":"2024-03-30T13:01:41.983624+0000","last_unstale":"2024-03-30T13:02:45.164855+0000","last_undegraded":"2024-03-30T13:02:45.164855+0000","last_fullsized":"2024-03-30T13:02:45.164855+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:01:51.537497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:51.537497+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:55.057623+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":422,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-03-30T13:02:46.468168+0000","last_change":"2024-03-30T13:02:40.333877+0000","last_active":"2024-03-30T13:02:46.468168+0000","last_peered":"2024-03-30T13:02:46.468168+0000","last_clean":"2024-03-30T13:02:46.468168+0000","last_became_active":"2024-03-30T13:01:41.986001+0000","last_became_peered":"2024-03-30T13:01:41.986001+0000","last_unstale":"2024-03-30T13:02:46.468168+0000","last_undegraded":"2024-03-30T13:02:46.468168+0000","last_fullsized":"2024-03-30T13:02:46.468168+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:02:40.333589+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:02:40.333589+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:45.470373+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10453822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":240672973,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509530,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48592,"kb_used_data":17412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323248,"statfs":{"total":96636764160,"available":96587005952,"internally_reserved":0,"allocated":17829888,"data_stored":16518105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":156,"seq":670014898192,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53800,"kb_used_data":22108,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94318040,"statfs":{"total":96636764160,"available":96581672960,"internally_reserved":0,"allocated":22638592,"data_stored":21354013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4405,"internal_metadata":32435915},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":135,"seq":579820584986,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45952,"kb_used_data":13800,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94325888,"statfs":{"total":96636764160,"available":96589709312,"internally_reserved":0,"allocated":14131200,"data_stored":12853422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6047,"internal_metadata":32893025},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509531,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39820,"kb_used_data":8628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332020,"statfs":{"total":96636764160,"available":96595988480,"internally_reserved":0,"allocated":8835072,"data_stored":7523289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285616,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39144,"kb_used_data":7448,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94332696,"statfs":{"total":96636764160,"available":96596680704,"internally_reserved":0,"allocated":7626752,"data_stored":6328571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509531,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46992,"kb_used_data":15804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324848,"statfs":{"total":96636764160,"available":96588644352,"internally_reserved":0,"allocated":16183296,"data_stored":14871513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":180,"seq":773094113285,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44892,"kb_used_data":12300,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94326948,"statfs":{"total":96636764160,"available":96590794752,"internally_reserved":0,"allocated":12595200,"data_stored":11310621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7690,"internal_metadata":33350134},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574939,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41800,"kb_used_data":14780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330040,"statfs":{"total":96636764160,"available":96593960960,"internally_reserved":0,"allocated":15134720,"data_stored":13836539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:02:51.489 INFO:tasks.thrashosds.thrasher:Choosing PG 1.2 with acting set [7, 0, 4] 2024-03-30T13:02:51.489 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 2, 3, 6, 1, 4, 5, 7] 2024-03-30T13:02:51.490 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-30T13:02:51.490 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:02:51.546 INFO:tasks.ceph.osd.7:Stopped 2024-03-30T13:02:51.546 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 2, 3, 6, 1, 4, 5, 7] 2024-03-30T13:02:51.546 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 7 2024-03-30T13:02:51.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:51.660+0000 7f1763579640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:51.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:51.672+0000 7f1763579640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:51.851 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:02:51.848+0000 7fbb38f08640 -1 mon.a@0(leader).osd e197 definitely_dead 0 2024-03-30T13:02:52.522 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.7. 2024-03-30T13:02:52.553 INFO:tasks.thrashosds.thrasher:dead_osds=[7], live_osds=[0, 2, 3, 6, 1, 4, 5] 2024-03-30T13:02:52.553 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:02:52.554 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:02:52.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:52.668+0000 7ff8a9b76640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:52.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:02:52.680+0000 7ff8a9b76640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:02:52.851 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:02:52.851 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:02:52.867 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":411,"stamp":"2024-03-30T13:02:52.526263+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":240672973,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":316212,"kb_used_data":100100,"kb_used_omap":20,"kb_used_meta":215979,"kb_avail":660286668,"statfs":{"total":676457349120,"available":676133548032,"internally_reserved":0,"allocated":102502400,"data_stored":93340322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21231,"internal_metadata":221162769},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.048418"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":451,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:02:48.544439+0000","last_change":"2024-03-30T13:01:42.155603+0000","last_active":"2024-03-30T13:02:48.544439+0000","last_peered":"2024-03-30T13:02:48.544439+0000","last_clean":"2024-03-30T13:02:48.544439+0000","last_became_active":"2024-03-30T13:01:41.987359+0000","last_became_peered":"2024-03-30T13:01:41.987359+0000","last_unstale":"2024-03-30T13:02:48.544439+0000","last_undegraded":"2024-03-30T13:02:48.544439+0000","last_fullsized":"2024-03-30T13:02:48.544439+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:01:22.928753+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:22.928753+0000","last_clean_scrub_stamp":"2024-03-30T13:01:22.928753+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:51.214009+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18352980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":413,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-30T13:02:49.499709+0000","last_change":"2024-03-30T13:02:34.363323+0000","last_active":"2024-03-30T13:02:49.499709+0000","last_peered":"2024-03-30T13:02:49.499709+0000","last_clean":"2024-03-30T13:02:49.499709+0000","last_became_active":"2024-03-30T13:02:34.336789+0000","last_became_peered":"2024-03-30T13:02:34.336789+0000","last_unstale":"2024-03-30T13:02:49.499709+0000","last_undegraded":"2024-03-30T13:02:49.499709+0000","last_fullsized":"2024-03-30T13:02:49.499709+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:27.269301+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:02:27.269301+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:36.631791+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":38,"reported_epoch":192,"state":"stale+active+clean","last_fresh":"2024-03-30T13:02:44.445401+0000","last_change":"2024-03-30T13:02:34.415429+0000","last_active":"2024-03-30T13:02:44.445401+0000","last_peered":"2024-03-30T13:02:44.445401+0000","last_clean":"2024-03-30T13:02:44.445401+0000","last_became_active":"2024-03-30T13:02:34.331788+0000","last_became_peered":"2024-03-30T13:02:34.331788+0000","last_unstale":"2024-03-30T13:02:52.526290+0000","last_undegraded":"2024-03-30T13:02:44.445401+0000","last_fullsized":"2024-03-30T13:02:44.445401+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:02:33.302956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:33.302956+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:35.749650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":36,"reported_epoch":192,"state":"stale+active+clean","last_fresh":"2024-03-30T13:02:44.445256+0000","last_change":"2024-03-30T13:02:34.996784+0000","last_active":"2024-03-30T13:02:44.445256+0000","last_peered":"2024-03-30T13:02:44.445256+0000","last_clean":"2024-03-30T13:02:44.445256+0000","last_became_active":"2024-03-30T13:02:34.963470+0000","last_became_peered":"2024-03-30T13:02:34.963470+0000","last_unstale":"2024-03-30T13:02:52.526294+0000","last_undegraded":"2024-03-30T13:02:44.445256+0000","last_fullsized":"2024-03-30T13:02:44.445256+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:02:02.359931+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:02:02.359931+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:12.493904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":97,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-30T13:02:49.959613+0000","last_change":"2024-03-30T13:02:35.984755+0000","last_active":"2024-03-30T13:02:49.959613+0000","last_peered":"2024-03-30T13:02:49.959613+0000","last_clean":"2024-03-30T13:02:49.959613+0000","last_became_active":"2024-03-30T13:01:41.984313+0000","last_became_peered":"2024-03-30T13:01:41.984313+0000","last_unstale":"2024-03-30T13:02:49.959613+0000","last_undegraded":"2024-03-30T13:02:49.959613+0000","last_fullsized":"2024-03-30T13:02:49.959613+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:35.984506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:35.984506+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:55.717882+0000","scrub_duration":291,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":448,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-03-30T13:02:47.458812+0000","last_change":"2024-03-30T13:02:10.596835+0000","last_active":"2024-03-30T13:02:47.458812+0000","last_peered":"2024-03-30T13:02:47.458812+0000","last_clean":"2024-03-30T13:02:47.458812+0000","last_became_active":"2024-03-30T13:01:41.987920+0000","last_became_peered":"2024-03-30T13:01:41.987920+0000","last_unstale":"2024-03-30T13:02:47.458812+0000","last_undegraded":"2024-03-30T13:02:47.458812+0000","last_fullsized":"2024-03-30T13:02:47.458812+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:02:10.596458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:10.596458+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:38.665640+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22789962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":51,"reported_epoch":192,"state":"stale+active+clean","last_fresh":"2024-03-30T13:02:44.445345+0000","last_change":"2024-03-30T13:02:34.570436+0000","last_active":"2024-03-30T13:02:44.445345+0000","last_peered":"2024-03-30T13:02:44.445345+0000","last_clean":"2024-03-30T13:02:44.445345+0000","last_became_active":"2024-03-30T13:02:34.332732+0000","last_became_peered":"2024-03-30T13:02:34.332732+0000","last_unstale":"2024-03-30T13:02:52.526298+0000","last_undegraded":"2024-03-30T13:02:44.445345+0000","last_fullsized":"2024-03-30T13:02:44.445345+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:01:48.791509+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:01:48.791509+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:54.580526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":436,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-03-30T13:02:46.468050+0000","last_change":"2024-03-30T13:02:37.318261+0000","last_active":"2024-03-30T13:02:46.468050+0000","last_peered":"2024-03-30T13:02:46.468050+0000","last_clean":"2024-03-30T13:02:46.468050+0000","last_became_active":"2024-03-30T13:01:41.988151+0000","last_became_peered":"2024-03-30T13:01:41.988151+0000","last_unstale":"2024-03-30T13:02:46.468050+0000","last_undegraded":"2024-03-30T13:02:46.468050+0000","last_fullsized":"2024-03-30T13:02:46.468050+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:02:37.317994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:37.317994+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:02.570664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":22033680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":500,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-03-30T13:02:47.459234+0000","last_change":"2024-03-30T13:02:34.376265+0000","last_active":"2024-03-30T13:02:47.459234+0000","last_peered":"2024-03-30T13:02:47.459234+0000","last_clean":"2024-03-30T13:02:47.459234+0000","last_became_active":"2024-03-30T13:02:34.338628+0000","last_became_peered":"2024-03-30T13:02:34.338628+0000","last_unstale":"2024-03-30T13:02:47.459234+0000","last_undegraded":"2024-03-30T13:02:47.459234+0000","last_fullsized":"2024-03-30T13:02:47.459234+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:02:08.597557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:08.597557+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:23.617852+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":40605184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":331,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-30T13:02:49.499667+0000","last_change":"2024-03-30T13:02:28.264089+0000","last_active":"2024-03-30T13:02:49.499667+0000","last_peered":"2024-03-30T13:02:49.499667+0000","last_clean":"2024-03-30T13:02:49.499667+0000","last_became_active":"2024-03-30T13:00:46.109054+0000","last_became_peered":"2024-03-30T13:00:46.109054+0000","last_unstale":"2024-03-30T13:02:49.499667+0000","last_undegraded":"2024-03-30T13:02:49.499667+0000","last_fullsized":"2024-03-30T13:02:49.499667+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:28.263717+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:02:28.263717+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:56.652431+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":463,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:02:48.544397+0000","last_change":"2024-03-30T13:02:18.934000+0000","last_active":"2024-03-30T13:02:48.544397+0000","last_peered":"2024-03-30T13:02:48.544397+0000","last_clean":"2024-03-30T13:02:48.544397+0000","last_became_active":"2024-03-30T13:00:46.106211+0000","last_became_peered":"2024-03-30T13:00:46.106211+0000","last_unstale":"2024-03-30T13:02:48.544397+0000","last_undegraded":"2024-03-30T13:02:48.544397+0000","last_fullsized":"2024-03-30T13:02:48.544397+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:02:18.933526+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:42.090625+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25479082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":471,"reported_epoch":196,"state":"active+clean","last_fresh":"2024-03-30T13:02:48.544659+0000","last_change":"2024-03-30T13:02:24.920510+0000","last_active":"2024-03-30T13:02:48.544659+0000","last_peered":"2024-03-30T13:02:48.544659+0000","last_clean":"2024-03-30T13:02:48.544659+0000","last_became_active":"2024-03-30T13:01:41.991185+0000","last_became_peered":"2024-03-30T13:01:41.991185+0000","last_unstale":"2024-03-30T13:02:48.544659+0000","last_undegraded":"2024-03-30T13:02:48.544659+0000","last_fullsized":"2024-03-30T13:02:48.544659+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:02:24.920236+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T13:02:24.920236+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:28.373546+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":103,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-30T13:02:49.959574+0000","last_change":"2024-03-30T13:01:53.533883+0000","last_active":"2024-03-30T13:02:49.959574+0000","last_peered":"2024-03-30T13:02:49.959574+0000","last_clean":"2024-03-30T13:02:49.959574+0000","last_became_active":"2024-03-30T13:01:41.982566+0000","last_became_peered":"2024-03-30T13:01:41.982566+0000","last_unstale":"2024-03-30T13:02:49.959574+0000","last_undegraded":"2024-03-30T13:02:49.959574+0000","last_fullsized":"2024-03-30T13:02:49.959574+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:01:53.533647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:53.533647+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:00.040107+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":34,"reported_epoch":192,"state":"stale+active+clean","last_fresh":"2024-03-30T13:02:44.445304+0000","last_change":"2024-03-30T13:02:34.360717+0000","last_active":"2024-03-30T13:02:44.445304+0000","last_peered":"2024-03-30T13:02:44.445304+0000","last_clean":"2024-03-30T13:02:44.445304+0000","last_became_active":"2024-03-30T13:02:34.331686+0000","last_became_peered":"2024-03-30T13:02:34.331686+0000","last_unstale":"2024-03-30T13:02:52.526302+0000","last_undegraded":"2024-03-30T13:02:44.445304+0000","last_fullsized":"2024-03-30T13:02:44.445304+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:08.666884+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:02:08.666884+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:11.626036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":103,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-30T13:02:49.959613+0000","last_change":"2024-03-30T13:01:51.537905+0000","last_active":"2024-03-30T13:02:49.959613+0000","last_peered":"2024-03-30T13:02:49.959613+0000","last_clean":"2024-03-30T13:02:49.959613+0000","last_became_active":"2024-03-30T13:01:41.983624+0000","last_became_peered":"2024-03-30T13:01:41.983624+0000","last_unstale":"2024-03-30T13:02:49.959613+0000","last_undegraded":"2024-03-30T13:02:49.959613+0000","last_fullsized":"2024-03-30T13:02:49.959613+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:01:51.537497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:01:51.537497+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:02:55.057623+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":422,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-03-30T13:02:46.468168+0000","last_change":"2024-03-30T13:02:40.333877+0000","last_active":"2024-03-30T13:02:46.468168+0000","last_peered":"2024-03-30T13:02:46.468168+0000","last_clean":"2024-03-30T13:02:46.468168+0000","last_became_active":"2024-03-30T13:01:41.986001+0000","last_became_peered":"2024-03-30T13:01:41.986001+0000","last_unstale":"2024-03-30T13:02:46.468168+0000","last_undegraded":"2024-03-30T13:02:46.468168+0000","last_fullsized":"2024-03-30T13:02:46.468168+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:02:40.333589+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:02:40.333589+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:45.470373+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10453822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":240672973,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509530,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48592,"kb_used_data":17412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323248,"statfs":{"total":96636764160,"available":96587005952,"internally_reserved":0,"allocated":17829888,"data_stored":16518105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":156,"seq":670014898193,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53860,"kb_used_data":22168,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94317980,"statfs":{"total":96636764160,"available":96581611520,"internally_reserved":0,"allocated":22700032,"data_stored":21381448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4405,"internal_metadata":32435915},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":135,"seq":579820584987,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46004,"kb_used_data":13860,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94325836,"statfs":{"total":96636764160,"available":96589656064,"internally_reserved":0,"allocated":14192640,"data_stored":12880857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6047,"internal_metadata":32893025},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509531,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39820,"kb_used_data":8628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94332020,"statfs":{"total":96636764160,"available":96595988480,"internally_reserved":0,"allocated":8835072,"data_stored":7523289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285616,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39144,"kb_used_data":7448,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94332696,"statfs":{"total":96636764160,"available":96596680704,"internally_reserved":0,"allocated":7626752,"data_stored":6328571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509531,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46992,"kb_used_data":15804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324848,"statfs":{"total":96636764160,"available":96588644352,"internally_reserved":0,"allocated":16183296,"data_stored":14871513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574939,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41800,"kb_used_data":14780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94330040,"statfs":{"total":96636764160,"available":96593960960,"internally_reserved":0,"allocated":15134720,"data_stored":13836539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:02:52.868 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:03:17.869 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:03:17.869 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:03:18.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:17.992+0000 7f010a39f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:18.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:18.004+0000 7f010a39f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:18.182 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:03:18.183 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:03:18.201 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":430,"stamp":"2024-03-30T13:03:17.915292+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":248404094,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":326900,"kb_used_data":110804,"kb_used_omap":20,"kb_used_meta":215979,"kb_avail":660275980,"statfs":{"total":676457349120,"available":676122603520,"internally_reserved":0,"allocated":113463296,"data_stored":103930327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21231,"internal_metadata":221162769},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.834696"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":473,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-30T13:03:13.096213+0000","last_change":"2024-03-30T13:02:52.868231+0000","last_active":"2024-03-30T13:03:13.096213+0000","last_peered":"2024-03-30T13:03:13.096213+0000","last_clean":"2024-03-30T13:03:13.096213+0000","last_became_active":"2024-03-30T13:01:41.987359+0000","last_became_peered":"2024-03-30T13:01:41.987359+0000","last_unstale":"2024-03-30T13:03:13.096213+0000","last_undegraded":"2024-03-30T13:03:13.096213+0000","last_fullsized":"2024-03-30T13:03:13.096213+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:02:52.867981+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:22.928753+0000","last_clean_scrub_stamp":"2024-03-30T13:02:52.867981+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:09.169753+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18352980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":438,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-30T13:03:14.388209+0000","last_change":"2024-03-30T13:02:53.577196+0000","last_active":"2024-03-30T13:03:14.388209+0000","last_peered":"2024-03-30T13:03:14.388209+0000","last_clean":"2024-03-30T13:03:14.388209+0000","last_became_active":"2024-03-30T13:02:53.544668+0000","last_became_peered":"2024-03-30T13:02:53.544668+0000","last_unstale":"2024-03-30T13:03:14.388209+0000","last_undegraded":"2024-03-30T13:03:14.388209+0000","last_fullsized":"2024-03-30T13:03:14.388209+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:27.269301+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:02:27.269301+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:56.277337+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10621892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":29,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-30T13:03:14.117036+0000","last_change":"2024-03-30T13:02:53.619194+0000","last_active":"2024-03-30T13:03:14.117036+0000","last_peered":"2024-03-30T13:03:14.117036+0000","last_clean":"2024-03-30T13:03:14.117036+0000","last_became_active":"2024-03-30T13:02:53.544564+0000","last_became_peered":"2024-03-30T13:02:53.544564+0000","last_unstale":"2024-03-30T13:03:14.117036+0000","last_undegraded":"2024-03-30T13:03:14.117036+0000","last_fullsized":"2024-03-30T13:03:14.117036+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:02:33.302956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:33.302956+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:58.292872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":23,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-03-30T13:03:11.083171+0000","last_change":"2024-03-30T13:02:53.595573+0000","last_active":"2024-03-30T13:03:11.083171+0000","last_peered":"2024-03-30T13:03:11.083171+0000","last_clean":"2024-03-30T13:03:11.083171+0000","last_became_active":"2024-03-30T13:02:53.539885+0000","last_became_peered":"2024-03-30T13:02:53.539885+0000","last_unstale":"2024-03-30T13:03:11.083171+0000","last_undegraded":"2024-03-30T13:03:11.083171+0000","last_fullsized":"2024-03-30T13:03:11.083171+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:02:02.359931+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:02:02.359931+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:25.387833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":111,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-30T13:03:14.113444+0000","last_change":"2024-03-30T13:02:35.984755+0000","last_active":"2024-03-30T13:03:14.113444+0000","last_peered":"2024-03-30T13:03:14.113444+0000","last_clean":"2024-03-30T13:03:14.113444+0000","last_became_active":"2024-03-30T13:01:41.984313+0000","last_became_peered":"2024-03-30T13:01:41.984313+0000","last_unstale":"2024-03-30T13:03:14.113444+0000","last_undegraded":"2024-03-30T13:03:14.113444+0000","last_fullsized":"2024-03-30T13:03:14.113444+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:35.984506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:35.984506+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:55.717882+0000","scrub_duration":291,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":467,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-30T13:03:14.450048+0000","last_change":"2024-03-30T13:02:10.596835+0000","last_active":"2024-03-30T13:03:14.450048+0000","last_peered":"2024-03-30T13:03:14.450048+0000","last_clean":"2024-03-30T13:03:14.450048+0000","last_became_active":"2024-03-30T13:01:41.987920+0000","last_became_peered":"2024-03-30T13:01:41.987920+0000","last_unstale":"2024-03-30T13:03:14.450048+0000","last_undegraded":"2024-03-30T13:03:14.450048+0000","last_fullsized":"2024-03-30T13:03:14.450048+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:02:10.596458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:10.596458+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:38.665640+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22789962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":50,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-30T13:03:14.388081+0000","last_change":"2024-03-30T13:03:13.095455+0000","last_active":"2024-03-30T13:03:14.388081+0000","last_peered":"2024-03-30T13:03:14.388081+0000","last_clean":"2024-03-30T13:03:14.388081+0000","last_became_active":"2024-03-30T13:02:53.544642+0000","last_became_peered":"2024-03-30T13:02:53.544642+0000","last_unstale":"2024-03-30T13:03:14.388081+0000","last_undegraded":"2024-03-30T13:03:14.388081+0000","last_fullsized":"2024-03-30T13:03:14.388081+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:03:13.094988+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:03:13.094988+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:35.543167+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":452,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-03-30T13:03:11.083202+0000","last_change":"2024-03-30T13:02:37.318261+0000","last_active":"2024-03-30T13:03:11.083202+0000","last_peered":"2024-03-30T13:03:11.083202+0000","last_clean":"2024-03-30T13:03:11.083202+0000","last_became_active":"2024-03-30T13:01:41.988151+0000","last_became_peered":"2024-03-30T13:01:41.988151+0000","last_unstale":"2024-03-30T13:03:11.083202+0000","last_undegraded":"2024-03-30T13:03:11.083202+0000","last_fullsized":"2024-03-30T13:03:11.083202+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:02:37.317994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:37.317994+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:02.570664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":22033680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":531,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-30T13:03:14.450128+0000","last_change":"2024-03-30T13:02:53.687772+0000","last_active":"2024-03-30T13:03:14.450128+0000","last_peered":"2024-03-30T13:03:14.450128+0000","last_clean":"2024-03-30T13:03:14.450128+0000","last_became_active":"2024-03-30T13:02:53.544618+0000","last_became_peered":"2024-03-30T13:02:53.544618+0000","last_unstale":"2024-03-30T13:03:14.450128+0000","last_undegraded":"2024-03-30T13:03:14.450128+0000","last_fullsized":"2024-03-30T13:03:14.450128+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:02:08.597557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:08.597557+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:31.649911+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":45680832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":345,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-30T13:03:14.388102+0000","last_change":"2024-03-30T13:02:28.264089+0000","last_active":"2024-03-30T13:03:14.388102+0000","last_peered":"2024-03-30T13:03:14.388102+0000","last_clean":"2024-03-30T13:03:14.388102+0000","last_became_active":"2024-03-30T13:00:46.109054+0000","last_became_peered":"2024-03-30T13:00:46.109054+0000","last_unstale":"2024-03-30T13:03:14.388102+0000","last_undegraded":"2024-03-30T13:03:14.388102+0000","last_fullsized":"2024-03-30T13:03:14.388102+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:28.263717+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:02:28.263717+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:56.652431+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":477,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-30T13:03:13.096136+0000","last_change":"2024-03-30T13:02:18.934000+0000","last_active":"2024-03-30T13:03:13.096136+0000","last_peered":"2024-03-30T13:03:13.096136+0000","last_clean":"2024-03-30T13:03:13.096136+0000","last_became_active":"2024-03-30T13:00:46.106211+0000","last_became_peered":"2024-03-30T13:00:46.106211+0000","last_unstale":"2024-03-30T13:03:13.096136+0000","last_undegraded":"2024-03-30T13:03:13.096136+0000","last_fullsized":"2024-03-30T13:03:13.096136+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:02:18.933526+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:42.090625+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25479082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":485,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-30T13:03:13.096514+0000","last_change":"2024-03-30T13:02:24.920510+0000","last_active":"2024-03-30T13:03:13.096514+0000","last_peered":"2024-03-30T13:03:13.096514+0000","last_clean":"2024-03-30T13:03:13.096514+0000","last_became_active":"2024-03-30T13:01:41.991185+0000","last_became_peered":"2024-03-30T13:01:41.991185+0000","last_unstale":"2024-03-30T13:03:13.096514+0000","last_undegraded":"2024-03-30T13:03:13.096514+0000","last_fullsized":"2024-03-30T13:03:13.096514+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:02:24.920236+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T12:57:11.126678+0000","last_clean_scrub_stamp":"2024-03-30T13:02:24.920236+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:28.373546+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":125,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-30T13:03:14.113331+0000","last_change":"2024-03-30T13:03:01.546561+0000","last_active":"2024-03-30T13:03:14.113331+0000","last_peered":"2024-03-30T13:03:14.113331+0000","last_clean":"2024-03-30T13:03:14.113331+0000","last_became_active":"2024-03-30T13:01:41.982566+0000","last_became_peered":"2024-03-30T13:01:41.982566+0000","last_unstale":"2024-03-30T13:03:14.113331+0000","last_undegraded":"2024-03-30T13:03:14.113331+0000","last_fullsized":"2024-03-30T13:03:14.113331+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:03:01.546292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:01.546292+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:20.359919+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":25,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-30T13:03:14.113370+0000","last_change":"2024-03-30T13:02:53.578947+0000","last_active":"2024-03-30T13:03:14.113370+0000","last_peered":"2024-03-30T13:03:14.113370+0000","last_clean":"2024-03-30T13:03:14.113370+0000","last_became_active":"2024-03-30T13:02:53.543819+0000","last_became_peered":"2024-03-30T13:02:53.543819+0000","last_unstale":"2024-03-30T13:03:14.113370+0000","last_undegraded":"2024-03-30T13:03:14.113370+0000","last_fullsized":"2024-03-30T13:03:14.113370+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:08.666884+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:02:08.666884+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:31.498438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":125,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-03-30T13:03:14.113473+0000","last_change":"2024-03-30T13:02:56.617180+0000","last_active":"2024-03-30T13:03:14.113473+0000","last_peered":"2024-03-30T13:03:14.113473+0000","last_clean":"2024-03-30T13:03:14.113473+0000","last_became_active":"2024-03-30T13:01:41.983624+0000","last_became_peered":"2024-03-30T13:01:41.983624+0000","last_unstale":"2024-03-30T13:03:14.113473+0000","last_undegraded":"2024-03-30T13:03:14.113473+0000","last_fullsized":"2024-03-30T13:03:14.113473+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:02:56.616926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:56.616926+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:25.091190+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":437,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-03-30T13:03:11.083191+0000","last_change":"2024-03-30T13:02:40.333877+0000","last_active":"2024-03-30T13:03:11.083191+0000","last_peered":"2024-03-30T13:03:11.083191+0000","last_clean":"2024-03-30T13:03:11.083191+0000","last_became_active":"2024-03-30T13:01:41.986001+0000","last_became_peered":"2024-03-30T13:01:41.986001+0000","last_unstale":"2024-03-30T13:03:11.083191+0000","last_undegraded":"2024-03-30T13:03:11.083191+0000","last_fullsized":"2024-03-30T13:03:11.083191+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:02:40.333589+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:02:40.333589+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:45.470373+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10453822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":248404094,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106594304,"data_stored":106594304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509536,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48680,"kb_used_data":17496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323160,"statfs":{"total":96636764160,"available":96586915840,"internally_reserved":0,"allocated":17915904,"data_stored":16556948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":156,"seq":670014898198,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54588,"kb_used_data":22892,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94317252,"statfs":{"total":96636764160,"available":96580866048,"internally_reserved":0,"allocated":23441408,"data_stored":22075693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4405,"internal_metadata":32435915},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":135,"seq":579820584992,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46096,"kb_used_data":13956,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94325744,"statfs":{"total":96636764160,"available":96589561856,"internally_reserved":0,"allocated":14290944,"data_stored":12925229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6047,"internal_metadata":32893025},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509536,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46000,"kb_used_data":14820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325840,"statfs":{"total":96636764160,"available":96589660160,"internally_reserved":0,"allocated":15175680,"data_stored":13809965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285621,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42544,"kb_used_data":10852,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94329296,"statfs":{"total":96636764160,"available":96593199104,"internally_reserved":0,"allocated":11112448,"data_stored":9760251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509536,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47068,"kb_used_data":15888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324772,"statfs":{"total":96636764160,"available":96588566528,"internally_reserved":0,"allocated":16269312,"data_stored":14910356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574945,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41924,"kb_used_data":14900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329916,"statfs":{"total":96636764160,"available":96593833984,"internally_reserved":0,"allocated":15257600,"data_stored":13891885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8679424,"data_stored":8679424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12718080,"data_stored":12718080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20983808,"data_stored":20983808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:03:18.202 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:03:18.202 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:03:18.202 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2024-03-30T13:03:18.202 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-30T13:03:18.202 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-30T13:03:18.205 INFO:tasks.ceph.osd.7:Started 2024-03-30T13:03:18.205 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-30T13:03:18.269 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:03:18.266+0000 7f7a2e00e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:18.276 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:03:18.274+0000 7f7a2e00e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:18.276 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:03:18.274+0000 7f7a2e00e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:18.322 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T13:03:18.323 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T13:03:18.323 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T13:03:18.323 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T13:03:18.323 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T13:03:18.323 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T13:03:18.323 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-03-30T13:03:18.323 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-03-30T13:03:18.323 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-03-30T13:03:18.323 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-03-30T13:03:18.323 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-03-30T13:03:18.323 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T13:03:18.325 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:03:18.325 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-03-30T13:03:18.815 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:03:18.814+0000 7f7a2e00e7c0 -1 Falling back to public interface 2024-03-30T13:03:22.545 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:03:22.542+0000 7f7a2e00e7c0 -1 osd.7 197 log_to_monitors true 2024-03-30T13:03:23.170 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:03:23.170+0000 7f7a25de3640 -1 osd.7 197 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:03:23.327 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-30T13:03:23.457 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T13:03:23.457 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T13:03:23.457 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T13:03:23.457 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T13:03:23.471 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:03:23.471 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-03-30T13:03:23.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 7 2024-03-30T13:03:23.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:23.596+0000 7f873c811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:23.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:23.612+0000 7f873c811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:23.792 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:03:23.788+0000 7fbb38f08640 -1 mon.a@0(leader).osd e205 definitely_dead 0 2024-03-30T13:03:24.148 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.7. 2024-03-30T13:03:24.180 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-03-30T13:03:24.180 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:03:24.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:24.284+0000 7f3193ca1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:24.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:24.300+0000 7f3193ca1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:25.170 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:03:25.200 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:03:25.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:25.312+0000 7f5240d46640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:25.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:25.332+0000 7f5240d46640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:26.194 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:03:26.220 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:03:26.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:26.332+0000 7f4aae825640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:26.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:26.348+0000 7f4aae825640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:27.190 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:03:27.216 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:03:27.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:27.336+0000 7f7424402640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:27.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:27.352+0000 7f7424402640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:28.202 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:03:28.227 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:03:28.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:28.335+0000 7f7929e34640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:28.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:28.351+0000 7f7929e34640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:29.214 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:03:29.244 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:03:29.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:29.363+0000 7f55f57a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:29.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:29.379+0000 7f55f57a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:30.234 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:03:30.259 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:03:30.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:30.375+0000 7f8e070e8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:30.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:30.391+0000 7f8e070e8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:31.242 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:03:31.266 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:03:31.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:31.383+0000 7f9d80862640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:31.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:31.399+0000 7f9d80862640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:32.252 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:03:32.277 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:03:32.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:32.391+0000 7fa86f103640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:32.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:32.407+0000 7fa86f103640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:33.262 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:03:33.289 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:03:33.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:33.407+0000 7f79624a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:33.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:33.423+0000 7f79624a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:34.267 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:03:34.292 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:03:34.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:34.407+0000 7fccad3f4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:34.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:34.427+0000 7fccad3f4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:35.278 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:03:35.304 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:03:35.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:35.415+0000 7fc6b7858640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:35.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:35.431+0000 7fc6b7858640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:36.298 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:03:36.322 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:03:36.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:36.439+0000 7fbcbea32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:36.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:36.455+0000 7fbcbea32640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:37.308 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:03:37.334 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:03:37.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:37.455+0000 7fc7f75c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:37.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:37.471+0000 7fc7f75c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:38.319 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:03:38.344 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:03:38.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:38.459+0000 7f8a8c835640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:38.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:38.475+0000 7f8a8c835640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:39.321 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:03:39.348 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:03:39.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:39.463+0000 7f34c7025640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:39.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:39.479+0000 7f34c7025640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:40.349 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:03:40.375 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:03:40.375 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:03:40.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:40.491+0000 7f7f9182f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:40.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:40.503+0000 7f7f9182f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:40.680 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:03:40.681 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:03:40.698 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-03-30T13:03:40.341521+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":233513290,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":363828,"kb_used_data":114656,"kb_used_omap":29,"kb_used_meta":248994,"kb_avail":754610892,"statfs":{"total":773094113280,"available":772721553408,"internally_reserved":0,"allocated":117407744,"data_stored":105699179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30569,"internal_metadata":254970007},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.049657"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":507,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-30T13:03:39.554648+0000","last_change":"2024-03-30T13:02:52.868231+0000","last_active":"2024-03-30T13:03:39.554648+0000","last_peered":"2024-03-30T13:03:39.554648+0000","last_clean":"2024-03-30T13:03:39.554648+0000","last_became_active":"2024-03-30T13:01:41.987359+0000","last_became_peered":"2024-03-30T13:01:41.987359+0000","last_unstale":"2024-03-30T13:03:39.554648+0000","last_undegraded":"2024-03-30T13:03:39.554648+0000","last_fullsized":"2024-03-30T13:03:39.554648+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:02:52.867981+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:22.928753+0000","last_clean_scrub_stamp":"2024-03-30T13:02:52.867981+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:09.169753+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18352980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":478,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-30T13:03:39.326382+0000","last_change":"2024-03-30T13:03:25.182999+0000","last_active":"2024-03-30T13:03:39.326382+0000","last_peered":"2024-03-30T13:03:39.326382+0000","last_clean":"2024-03-30T13:03:39.326382+0000","last_became_active":"2024-03-30T13:03:25.181879+0000","last_became_peered":"2024-03-30T13:03:25.181879+0000","last_unstale":"2024-03-30T13:03:39.326382+0000","last_undegraded":"2024-03-30T13:03:39.326382+0000","last_fullsized":"2024-03-30T13:03:39.326382+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:27.269301+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:02:27.269301+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:43.734307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10621892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":55,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-30T13:03:38.972258+0000","last_change":"2024-03-30T13:03:25.176251+0000","last_active":"2024-03-30T13:03:38.972258+0000","last_peered":"2024-03-30T13:03:38.972258+0000","last_clean":"2024-03-30T13:03:38.972258+0000","last_became_active":"2024-03-30T13:03:25.175168+0000","last_became_peered":"2024-03-30T13:03:25.175168+0000","last_unstale":"2024-03-30T13:03:38.972258+0000","last_undegraded":"2024-03-30T13:03:38.972258+0000","last_fullsized":"2024-03-30T13:03:38.972258+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:02:33.302956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:33.302956+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":61,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-30T13:03:38.971928+0000","last_change":"2024-03-30T13:03:29.559445+0000","last_active":"2024-03-30T13:03:38.971928+0000","last_peered":"2024-03-30T13:03:38.971928+0000","last_clean":"2024-03-30T13:03:38.971928+0000","last_became_active":"2024-03-30T13:03:25.178075+0000","last_became_peered":"2024-03-30T13:03:25.178075+0000","last_unstale":"2024-03-30T13:03:38.971928+0000","last_undegraded":"2024-03-30T13:03:38.971928+0000","last_fullsized":"2024-03-30T13:03:38.971928+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:03:29.559074+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:03:29.559074+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:50.935813+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":136,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:03:35.353807+0000","last_change":"2024-03-30T13:02:35.984755+0000","last_active":"2024-03-30T13:03:35.353807+0000","last_peered":"2024-03-30T13:03:35.353807+0000","last_clean":"2024-03-30T13:03:35.353807+0000","last_became_active":"2024-03-30T13:01:41.984313+0000","last_became_peered":"2024-03-30T13:01:41.984313+0000","last_unstale":"2024-03-30T13:03:35.353807+0000","last_undegraded":"2024-03-30T13:03:35.353807+0000","last_fullsized":"2024-03-30T13:03:35.353807+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:35.984506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:35.984506+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:55.717882+0000","scrub_duration":291,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":495,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-03-30T13:03:36.365479+0000","last_change":"2024-03-30T13:02:10.596835+0000","last_active":"2024-03-30T13:03:36.365479+0000","last_peered":"2024-03-30T13:03:36.365479+0000","last_clean":"2024-03-30T13:03:36.365479+0000","last_became_active":"2024-03-30T13:01:41.987920+0000","last_became_peered":"2024-03-30T13:01:41.987920+0000","last_unstale":"2024-03-30T13:03:36.365479+0000","last_undegraded":"2024-03-30T13:03:36.365479+0000","last_fullsized":"2024-03-30T13:03:36.365479+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:02:10.596458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:10.596458+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:38.665640+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22789962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":82,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-30T13:03:38.971958+0000","last_change":"2024-03-30T13:03:25.276567+0000","last_active":"2024-03-30T13:03:38.971958+0000","last_peered":"2024-03-30T13:03:38.971958+0000","last_clean":"2024-03-30T13:03:38.971958+0000","last_became_active":"2024-03-30T13:03:25.178475+0000","last_became_peered":"2024-03-30T13:03:25.178475+0000","last_unstale":"2024-03-30T13:03:38.971958+0000","last_undegraded":"2024-03-30T13:03:38.971958+0000","last_fullsized":"2024-03-30T13:03:38.971958+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:03:13.094988+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:03:13.094988+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:21.173842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":486,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-30T13:03:37.307252+0000","last_change":"2024-03-30T13:02:37.318261+0000","last_active":"2024-03-30T13:03:37.307252+0000","last_peered":"2024-03-30T13:03:37.307252+0000","last_clean":"2024-03-30T13:03:37.307252+0000","last_became_active":"2024-03-30T13:01:41.988151+0000","last_became_peered":"2024-03-30T13:01:41.988151+0000","last_unstale":"2024-03-30T13:03:37.307252+0000","last_undegraded":"2024-03-30T13:03:37.307252+0000","last_fullsized":"2024-03-30T13:03:37.307252+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:02:37.317994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:37.317994+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:02.570664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":22033680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":573,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-03-30T13:03:36.365742+0000","last_change":"2024-03-30T13:03:30.573752+0000","last_active":"2024-03-30T13:03:36.365742+0000","last_peered":"2024-03-30T13:03:36.365742+0000","last_clean":"2024-03-30T13:03:36.365742+0000","last_became_active":"2024-03-30T13:03:25.980366+0000","last_became_peered":"2024-03-30T13:03:25.980366+0000","last_unstale":"2024-03-30T13:03:36.365742+0000","last_undegraded":"2024-03-30T13:03:36.365742+0000","last_fullsized":"2024-03-30T13:03:36.365742+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:03:30.573489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:30.573489+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:39.840213+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":45680832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":379,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-30T13:03:39.326255+0000","last_change":"2024-03-30T13:02:28.264089+0000","last_active":"2024-03-30T13:03:39.326255+0000","last_peered":"2024-03-30T13:03:39.326255+0000","last_clean":"2024-03-30T13:03:39.326255+0000","last_became_active":"2024-03-30T13:00:46.109054+0000","last_became_peered":"2024-03-30T13:00:46.109054+0000","last_unstale":"2024-03-30T13:03:39.326255+0000","last_undegraded":"2024-03-30T13:03:39.326255+0000","last_fullsized":"2024-03-30T13:03:39.326255+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:28.263717+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:02:28.263717+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:56.652431+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":511,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-30T13:03:39.554687+0000","last_change":"2024-03-30T13:02:18.934000+0000","last_active":"2024-03-30T13:03:39.554687+0000","last_peered":"2024-03-30T13:03:39.554687+0000","last_clean":"2024-03-30T13:03:39.554687+0000","last_became_active":"2024-03-30T13:00:46.106211+0000","last_became_peered":"2024-03-30T13:00:46.106211+0000","last_unstale":"2024-03-30T13:03:39.554687+0000","last_undegraded":"2024-03-30T13:03:39.554687+0000","last_fullsized":"2024-03-30T13:03:39.554687+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:02:18.933526+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:42.090625+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25479082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":527,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-30T13:03:39.554759+0000","last_change":"2024-03-30T13:03:28.656036+0000","last_active":"2024-03-30T13:03:39.554759+0000","last_peered":"2024-03-30T13:03:39.554759+0000","last_clean":"2024-03-30T13:03:39.554759+0000","last_became_active":"2024-03-30T13:01:41.991185+0000","last_became_peered":"2024-03-30T13:01:41.991185+0000","last_unstale":"2024-03-30T13:03:39.554759+0000","last_undegraded":"2024-03-30T13:03:39.554759+0000","last_fullsized":"2024-03-30T13:03:39.554759+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:03:28.655753+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:46.701130+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":150,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:03:35.354449+0000","last_change":"2024-03-30T13:03:01.546561+0000","last_active":"2024-03-30T13:03:35.354449+0000","last_peered":"2024-03-30T13:03:35.354449+0000","last_clean":"2024-03-30T13:03:35.354449+0000","last_became_active":"2024-03-30T13:01:41.982566+0000","last_became_peered":"2024-03-30T13:01:41.982566+0000","last_unstale":"2024-03-30T13:03:35.354449+0000","last_undegraded":"2024-03-30T13:03:35.354449+0000","last_fullsized":"2024-03-30T13:03:35.354449+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:03:01.546292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:01.546292+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:20.359919+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":59,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-30T13:03:38.971928+0000","last_change":"2024-03-30T13:03:32.547779+0000","last_active":"2024-03-30T13:03:38.971928+0000","last_peered":"2024-03-30T13:03:38.971928+0000","last_clean":"2024-03-30T13:03:38.971928+0000","last_became_active":"2024-03-30T13:03:25.178234+0000","last_became_peered":"2024-03-30T13:03:25.178234+0000","last_unstale":"2024-03-30T13:03:38.971928+0000","last_undegraded":"2024-03-30T13:03:38.971928+0000","last_fullsized":"2024-03-30T13:03:38.971928+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:03:32.547475+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:03:32.547475+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:48.688284+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":150,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:03:35.354027+0000","last_change":"2024-03-30T13:02:56.617180+0000","last_active":"2024-03-30T13:03:35.354027+0000","last_peered":"2024-03-30T13:03:35.354027+0000","last_clean":"2024-03-30T13:03:35.354027+0000","last_became_active":"2024-03-30T13:01:41.983624+0000","last_became_peered":"2024-03-30T13:01:41.983624+0000","last_unstale":"2024-03-30T13:03:35.354027+0000","last_undegraded":"2024-03-30T13:03:35.354027+0000","last_fullsized":"2024-03-30T13:03:35.354027+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:02:56.616926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:56.616926+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:25.091190+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":472,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-30T13:03:37.307263+0000","last_change":"2024-03-30T13:02:40.333877+0000","last_active":"2024-03-30T13:03:37.307263+0000","last_peered":"2024-03-30T13:03:37.307263+0000","last_clean":"2024-03-30T13:03:37.307263+0000","last_became_active":"2024-03-30T13:01:41.986001+0000","last_became_peered":"2024-03-30T13:01:41.986001+0000","last_unstale":"2024-03-30T13:03:37.307263+0000","last_undegraded":"2024-03-30T13:03:37.307263+0000","last_fullsized":"2024-03-30T13:03:37.307263+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:02:40.333589+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:02:40.333589+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:45.470373+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10453822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":233513290,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509541,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48868,"kb_used_data":17688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322972,"statfs":{"total":96636764160,"available":96586723328,"internally_reserved":0,"allocated":18112512,"data_stored":16646391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":156,"seq":670014898202,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54100,"kb_used_data":22396,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94317740,"statfs":{"total":96636764160,"available":96581365760,"internally_reserved":0,"allocated":22933504,"data_stored":21487786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4405,"internal_metadata":32435915},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":135,"seq":579820584997,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46304,"kb_used_data":14160,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94325536,"statfs":{"total":96636764160,"available":96589348864,"internally_reserved":0,"allocated":14499840,"data_stored":13020117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6047,"internal_metadata":32893025},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509541,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40092,"kb_used_data":8904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331748,"statfs":{"total":96636764160,"available":96595709952,"internally_reserved":0,"allocated":9117696,"data_stored":7651575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285626,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39436,"kb_used_data":7736,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94332404,"statfs":{"total":96636764160,"available":96596381696,"internally_reserved":0,"allocated":7921664,"data_stored":6462344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509541,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47276,"kb_used_data":16092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324564,"statfs":{"total":96636764160,"available":96588353536,"internally_reserved":0,"allocated":16478208,"data_stored":15005286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":206,"seq":884763262981,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45656,"kb_used_data":12612,"kb_used_omap":9,"kb_used_meta":33014,"kb_avail":94326184,"statfs":{"total":96636764160,"available":96590012416,"internally_reserved":0,"allocated":12914688,"data_stored":11455368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9338,"internal_metadata":33807238},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574949,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42096,"kb_used_data":15068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329744,"statfs":{"total":96636764160,"available":96593657856,"internally_reserved":0,"allocated":15429632,"data_stored":13970312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:03:40.700 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:03:40.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:40.815+0000 7f7320beb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:40.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:40.827+0000 7f7320beb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:41.003 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:03:41.004 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:03:41.022 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-03-30T13:03:40.341521+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":233513290,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":363828,"kb_used_data":114656,"kb_used_omap":29,"kb_used_meta":248994,"kb_avail":754610892,"statfs":{"total":773094113280,"available":772721553408,"internally_reserved":0,"allocated":117407744,"data_stored":105699179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30569,"internal_metadata":254970007},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.049657"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":507,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-30T13:03:39.554648+0000","last_change":"2024-03-30T13:02:52.868231+0000","last_active":"2024-03-30T13:03:39.554648+0000","last_peered":"2024-03-30T13:03:39.554648+0000","last_clean":"2024-03-30T13:03:39.554648+0000","last_became_active":"2024-03-30T13:01:41.987359+0000","last_became_peered":"2024-03-30T13:01:41.987359+0000","last_unstale":"2024-03-30T13:03:39.554648+0000","last_undegraded":"2024-03-30T13:03:39.554648+0000","last_fullsized":"2024-03-30T13:03:39.554648+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:02:52.867981+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:22.928753+0000","last_clean_scrub_stamp":"2024-03-30T13:02:52.867981+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:09.169753+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18352980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":478,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-30T13:03:39.326382+0000","last_change":"2024-03-30T13:03:25.182999+0000","last_active":"2024-03-30T13:03:39.326382+0000","last_peered":"2024-03-30T13:03:39.326382+0000","last_clean":"2024-03-30T13:03:39.326382+0000","last_became_active":"2024-03-30T13:03:25.181879+0000","last_became_peered":"2024-03-30T13:03:25.181879+0000","last_unstale":"2024-03-30T13:03:39.326382+0000","last_undegraded":"2024-03-30T13:03:39.326382+0000","last_fullsized":"2024-03-30T13:03:39.326382+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:27.269301+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:02:27.269301+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:43.734307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10621892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":55,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-30T13:03:38.972258+0000","last_change":"2024-03-30T13:03:25.176251+0000","last_active":"2024-03-30T13:03:38.972258+0000","last_peered":"2024-03-30T13:03:38.972258+0000","last_clean":"2024-03-30T13:03:38.972258+0000","last_became_active":"2024-03-30T13:03:25.175168+0000","last_became_peered":"2024-03-30T13:03:25.175168+0000","last_unstale":"2024-03-30T13:03:38.972258+0000","last_undegraded":"2024-03-30T13:03:38.972258+0000","last_fullsized":"2024-03-30T13:03:38.972258+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:02:33.302956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:33.302956+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":61,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-30T13:03:38.971928+0000","last_change":"2024-03-30T13:03:29.559445+0000","last_active":"2024-03-30T13:03:38.971928+0000","last_peered":"2024-03-30T13:03:38.971928+0000","last_clean":"2024-03-30T13:03:38.971928+0000","last_became_active":"2024-03-30T13:03:25.178075+0000","last_became_peered":"2024-03-30T13:03:25.178075+0000","last_unstale":"2024-03-30T13:03:38.971928+0000","last_undegraded":"2024-03-30T13:03:38.971928+0000","last_fullsized":"2024-03-30T13:03:38.971928+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:03:29.559074+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:03:29.559074+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:50.935813+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":136,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:03:35.353807+0000","last_change":"2024-03-30T13:02:35.984755+0000","last_active":"2024-03-30T13:03:35.353807+0000","last_peered":"2024-03-30T13:03:35.353807+0000","last_clean":"2024-03-30T13:03:35.353807+0000","last_became_active":"2024-03-30T13:01:41.984313+0000","last_became_peered":"2024-03-30T13:01:41.984313+0000","last_unstale":"2024-03-30T13:03:35.353807+0000","last_undegraded":"2024-03-30T13:03:35.353807+0000","last_fullsized":"2024-03-30T13:03:35.353807+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:35.984506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:35.984506+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:55.717882+0000","scrub_duration":291,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":495,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-03-30T13:03:36.365479+0000","last_change":"2024-03-30T13:02:10.596835+0000","last_active":"2024-03-30T13:03:36.365479+0000","last_peered":"2024-03-30T13:03:36.365479+0000","last_clean":"2024-03-30T13:03:36.365479+0000","last_became_active":"2024-03-30T13:01:41.987920+0000","last_became_peered":"2024-03-30T13:01:41.987920+0000","last_unstale":"2024-03-30T13:03:36.365479+0000","last_undegraded":"2024-03-30T13:03:36.365479+0000","last_fullsized":"2024-03-30T13:03:36.365479+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:02:10.596458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:10.596458+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:38.665640+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22789962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":82,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-30T13:03:38.971958+0000","last_change":"2024-03-30T13:03:25.276567+0000","last_active":"2024-03-30T13:03:38.971958+0000","last_peered":"2024-03-30T13:03:38.971958+0000","last_clean":"2024-03-30T13:03:38.971958+0000","last_became_active":"2024-03-30T13:03:25.178475+0000","last_became_peered":"2024-03-30T13:03:25.178475+0000","last_unstale":"2024-03-30T13:03:38.971958+0000","last_undegraded":"2024-03-30T13:03:38.971958+0000","last_fullsized":"2024-03-30T13:03:38.971958+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:03:13.094988+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:03:13.094988+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:21.173842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":486,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-30T13:03:37.307252+0000","last_change":"2024-03-30T13:02:37.318261+0000","last_active":"2024-03-30T13:03:37.307252+0000","last_peered":"2024-03-30T13:03:37.307252+0000","last_clean":"2024-03-30T13:03:37.307252+0000","last_became_active":"2024-03-30T13:01:41.988151+0000","last_became_peered":"2024-03-30T13:01:41.988151+0000","last_unstale":"2024-03-30T13:03:37.307252+0000","last_undegraded":"2024-03-30T13:03:37.307252+0000","last_fullsized":"2024-03-30T13:03:37.307252+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:02:37.317994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:37.317994+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:02.570664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":22033680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":573,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-03-30T13:03:36.365742+0000","last_change":"2024-03-30T13:03:30.573752+0000","last_active":"2024-03-30T13:03:36.365742+0000","last_peered":"2024-03-30T13:03:36.365742+0000","last_clean":"2024-03-30T13:03:36.365742+0000","last_became_active":"2024-03-30T13:03:25.980366+0000","last_became_peered":"2024-03-30T13:03:25.980366+0000","last_unstale":"2024-03-30T13:03:36.365742+0000","last_undegraded":"2024-03-30T13:03:36.365742+0000","last_fullsized":"2024-03-30T13:03:36.365742+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:03:30.573489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:30.573489+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:39.840213+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":45680832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":379,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-30T13:03:39.326255+0000","last_change":"2024-03-30T13:02:28.264089+0000","last_active":"2024-03-30T13:03:39.326255+0000","last_peered":"2024-03-30T13:03:39.326255+0000","last_clean":"2024-03-30T13:03:39.326255+0000","last_became_active":"2024-03-30T13:00:46.109054+0000","last_became_peered":"2024-03-30T13:00:46.109054+0000","last_unstale":"2024-03-30T13:03:39.326255+0000","last_undegraded":"2024-03-30T13:03:39.326255+0000","last_fullsized":"2024-03-30T13:03:39.326255+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:28.263717+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:02:28.263717+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:56.652431+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":511,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-30T13:03:39.554687+0000","last_change":"2024-03-30T13:02:18.934000+0000","last_active":"2024-03-30T13:03:39.554687+0000","last_peered":"2024-03-30T13:03:39.554687+0000","last_clean":"2024-03-30T13:03:39.554687+0000","last_became_active":"2024-03-30T13:00:46.106211+0000","last_became_peered":"2024-03-30T13:00:46.106211+0000","last_unstale":"2024-03-30T13:03:39.554687+0000","last_undegraded":"2024-03-30T13:03:39.554687+0000","last_fullsized":"2024-03-30T13:03:39.554687+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:02:18.933526+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:42.090625+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25479082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":527,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-30T13:03:39.554759+0000","last_change":"2024-03-30T13:03:28.656036+0000","last_active":"2024-03-30T13:03:39.554759+0000","last_peered":"2024-03-30T13:03:39.554759+0000","last_clean":"2024-03-30T13:03:39.554759+0000","last_became_active":"2024-03-30T13:01:41.991185+0000","last_became_peered":"2024-03-30T13:01:41.991185+0000","last_unstale":"2024-03-30T13:03:39.554759+0000","last_undegraded":"2024-03-30T13:03:39.554759+0000","last_fullsized":"2024-03-30T13:03:39.554759+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:03:28.655753+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:46.701130+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":150,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:03:35.354449+0000","last_change":"2024-03-30T13:03:01.546561+0000","last_active":"2024-03-30T13:03:35.354449+0000","last_peered":"2024-03-30T13:03:35.354449+0000","last_clean":"2024-03-30T13:03:35.354449+0000","last_became_active":"2024-03-30T13:01:41.982566+0000","last_became_peered":"2024-03-30T13:01:41.982566+0000","last_unstale":"2024-03-30T13:03:35.354449+0000","last_undegraded":"2024-03-30T13:03:35.354449+0000","last_fullsized":"2024-03-30T13:03:35.354449+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:03:01.546292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:01.546292+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:20.359919+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":59,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-30T13:03:38.971928+0000","last_change":"2024-03-30T13:03:32.547779+0000","last_active":"2024-03-30T13:03:38.971928+0000","last_peered":"2024-03-30T13:03:38.971928+0000","last_clean":"2024-03-30T13:03:38.971928+0000","last_became_active":"2024-03-30T13:03:25.178234+0000","last_became_peered":"2024-03-30T13:03:25.178234+0000","last_unstale":"2024-03-30T13:03:38.971928+0000","last_undegraded":"2024-03-30T13:03:38.971928+0000","last_fullsized":"2024-03-30T13:03:38.971928+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:03:32.547475+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:03:32.547475+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:48.688284+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":150,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-03-30T13:03:35.354027+0000","last_change":"2024-03-30T13:02:56.617180+0000","last_active":"2024-03-30T13:03:35.354027+0000","last_peered":"2024-03-30T13:03:35.354027+0000","last_clean":"2024-03-30T13:03:35.354027+0000","last_became_active":"2024-03-30T13:01:41.983624+0000","last_became_peered":"2024-03-30T13:01:41.983624+0000","last_unstale":"2024-03-30T13:03:35.354027+0000","last_undegraded":"2024-03-30T13:03:35.354027+0000","last_fullsized":"2024-03-30T13:03:35.354027+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:02:56.616926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:56.616926+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:25.091190+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":472,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-30T13:03:37.307263+0000","last_change":"2024-03-30T13:02:40.333877+0000","last_active":"2024-03-30T13:03:37.307263+0000","last_peered":"2024-03-30T13:03:37.307263+0000","last_clean":"2024-03-30T13:03:37.307263+0000","last_became_active":"2024-03-30T13:01:41.986001+0000","last_became_peered":"2024-03-30T13:01:41.986001+0000","last_unstale":"2024-03-30T13:03:37.307263+0000","last_undegraded":"2024-03-30T13:03:37.307263+0000","last_fullsized":"2024-03-30T13:03:37.307263+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:02:40.333589+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:02:40.333589+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:45.470373+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10453822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":233513290,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509541,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48868,"kb_used_data":17688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322972,"statfs":{"total":96636764160,"available":96586723328,"internally_reserved":0,"allocated":18112512,"data_stored":16646391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":156,"seq":670014898202,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54100,"kb_used_data":22396,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94317740,"statfs":{"total":96636764160,"available":96581365760,"internally_reserved":0,"allocated":22933504,"data_stored":21487786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4405,"internal_metadata":32435915},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":135,"seq":579820584997,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46304,"kb_used_data":14160,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94325536,"statfs":{"total":96636764160,"available":96589348864,"internally_reserved":0,"allocated":14499840,"data_stored":13020117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6047,"internal_metadata":32893025},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509541,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40092,"kb_used_data":8904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331748,"statfs":{"total":96636764160,"available":96595709952,"internally_reserved":0,"allocated":9117696,"data_stored":7651575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285626,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39436,"kb_used_data":7736,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94332404,"statfs":{"total":96636764160,"available":96596381696,"internally_reserved":0,"allocated":7921664,"data_stored":6462344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509541,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47276,"kb_used_data":16092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324564,"statfs":{"total":96636764160,"available":96588353536,"internally_reserved":0,"allocated":16478208,"data_stored":15005286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":206,"seq":884763262981,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45656,"kb_used_data":12612,"kb_used_omap":9,"kb_used_meta":33014,"kb_avail":94326184,"statfs":{"total":96636764160,"available":96590012416,"internally_reserved":0,"allocated":12914688,"data_stored":11455368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9338,"internal_metadata":33807238},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574949,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42096,"kb_used_data":15068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329744,"statfs":{"total":96636764160,"available":96593657856,"internally_reserved":0,"allocated":15429632,"data_stored":13970312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:03:41.023 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:03:41.023 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:03:41.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:41.131+0000 7f20128ca640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:41.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:41.143+0000 7f20128ca640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:41.317 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:03:41.317 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":222,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:03:40.330137+0000","last_up_change":"2024-03-30T13:03:24.146067+0000","last_in_change":"2024-03-30T13:03:24.146067+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":77,"up_from":83,"up_thru":198,"down_at":78,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6818","nonce":3104224893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6821","nonce":3104224893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6828","nonce":3104224893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6825","nonce":3104224893}]},"public_addr":"172.21.15.67:6818/3104224893","cluster_addr":"172.21.15.67:6821/3104224893","heartbeat_back_addr":"172.21.15.67:6828/3104224893","heartbeat_front_addr":"172.21.15.67:6825/3104224893","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6801","nonce":1504348724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6803","nonce":1504348724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6807","nonce":1504348724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6805","nonce":1504348724}]},"public_addr":"172.21.15.67:6801/1504348724","cluster_addr":"172.21.15.67:6803/1504348724","heartbeat_back_addr":"172.21.15.67:6807/1504348724","heartbeat_front_addr":"172.21.15.67:6805/1504348724","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":36,"last_clean_end":125,"up_from":135,"up_thru":206,"down_at":126,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":2818291404},{"type":"v1","addr":"172.21.15.82:6803","nonce":2818291404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":2818291404},{"type":"v1","addr":"172.21.15.82:6807","nonce":2818291404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":2818291404},{"type":"v1","addr":"172.21.15.82:6815","nonce":2818291404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":2818291404},{"type":"v1","addr":"172.21.15.82:6811","nonce":2818291404}]},"public_addr":"172.21.15.82:6803/2818291404","cluster_addr":"172.21.15.82:6807/2818291404","heartbeat_back_addr":"172.21.15.82:6815/2818291404","heartbeat_front_addr":"172.21.15.82:6811/2818291404","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":151,"up_from":156,"up_thru":198,"down_at":152,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":3428934162},{"type":"v1","addr":"172.21.15.82:6817","nonce":3428934162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":3428934162},{"type":"v1","addr":"172.21.15.82:6819","nonce":3428934162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":3428934162},{"type":"v1","addr":"172.21.15.82:6823","nonce":3428934162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":3428934162},{"type":"v1","addr":"172.21.15.82:6821","nonce":3428934162}]},"public_addr":"172.21.15.82:6817/3428934162","cluster_addr":"172.21.15.82:6819/3428934162","heartbeat_back_addr":"172.21.15.82:6823/3428934162","heartbeat_front_addr":"172.21.15.82:6821/3428934162","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":157,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6801","nonce":968508548}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6806","nonce":968508548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6814","nonce":968508548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":968508548},{"type":"v1","addr":"172.21.15.82:6810","nonce":968508548}]},"public_addr":"172.21.15.82:6801/968508548","cluster_addr":"172.21.15.82:6806/968508548","heartbeat_back_addr":"172.21.15.82:6814/968508548","heartbeat_front_addr":"172.21.15.82:6810/968508548","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":180,"last_clean_end":197,"up_from":206,"up_thru":206,"down_at":198,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":1241244807},{"type":"v1","addr":"172.21.15.82:6825","nonce":1241244807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":1241244807},{"type":"v1","addr":"172.21.15.82:6827","nonce":1241244807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":1241244807},{"type":"v1","addr":"172.21.15.82:6831","nonce":1241244807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":1241244807},{"type":"v1","addr":"172.21.15.82:6829","nonce":1241244807}]},"public_addr":"172.21.15.82:6825/1241244807","cluster_addr":"172.21.15.82:6827/1241244807","heartbeat_back_addr":"172.21.15.82:6831/1241244807","heartbeat_front_addr":"172.21.15.82:6829/1241244807","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:03:10.204605+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:03:12.370486+0000","dead_epoch":78},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:03:10.908939+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:03:13.947730+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:03:26.003466+0000","dead_epoch":126},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:03:08.285965+0000","dead_epoch":152},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:03:06.949779+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:02:32.251172+0000","dead_epoch":198}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:03:41.335 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:03:41.335 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:03:41.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:41.435+0000 7f9d1a8c2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:41.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:41.455+0000 7f9d1a8c2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:41.624 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:03:41.643 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:03:41.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:41.759+0000 7fd5c81f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:41.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:41.975+0000 7fd5c6792640 0 --2- 172.21.15.67:0/3102620349 >> v2:172.21.15.67:3300/0 conn(0x7fd5c00ad660 0x7fd5c013c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-03-30T13:03:41.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:41.979+0000 7fd5c81f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:42.169 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:03:42.169 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:03:42.188 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:03:42.188 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:03:42.188 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:03:42.188 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:03:42.188 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 3, 6, 1, 4, 5, 7] 2024-03-30T13:03:42.188 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:03:42.188 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:03:42.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:42.291+0000 7f9d60b4d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:42.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:42.307+0000 7f9d60b4d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:42.489 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:03:42.489 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:03:42.507 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":460,"stamp":"2024-03-30T13:03:41.921917+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":233513290,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":363884,"kb_used_data":114716,"kb_used_omap":29,"kb_used_meta":248994,"kb_avail":754610836,"statfs":{"total":773094113280,"available":772721496064,"internally_reserved":0,"allocated":117469184,"data_stored":105726614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30569,"internal_metadata":254970007},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.618316"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":507,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-30T13:03:39.554648+0000","last_change":"2024-03-30T13:02:52.868231+0000","last_active":"2024-03-30T13:03:39.554648+0000","last_peered":"2024-03-30T13:03:39.554648+0000","last_clean":"2024-03-30T13:03:39.554648+0000","last_became_active":"2024-03-30T13:01:41.987359+0000","last_became_peered":"2024-03-30T13:01:41.987359+0000","last_unstale":"2024-03-30T13:03:39.554648+0000","last_undegraded":"2024-03-30T13:03:39.554648+0000","last_fullsized":"2024-03-30T13:03:39.554648+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:02:52.867981+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:22.928753+0000","last_clean_scrub_stamp":"2024-03-30T13:02:52.867981+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:09.169753+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18352980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":478,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-30T13:03:39.326382+0000","last_change":"2024-03-30T13:03:25.182999+0000","last_active":"2024-03-30T13:03:39.326382+0000","last_peered":"2024-03-30T13:03:39.326382+0000","last_clean":"2024-03-30T13:03:39.326382+0000","last_became_active":"2024-03-30T13:03:25.181879+0000","last_became_peered":"2024-03-30T13:03:25.181879+0000","last_unstale":"2024-03-30T13:03:39.326382+0000","last_undegraded":"2024-03-30T13:03:39.326382+0000","last_fullsized":"2024-03-30T13:03:39.326382+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:27.269301+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:02:27.269301+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:43.734307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10621892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":55,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-30T13:03:38.972258+0000","last_change":"2024-03-30T13:03:25.176251+0000","last_active":"2024-03-30T13:03:38.972258+0000","last_peered":"2024-03-30T13:03:38.972258+0000","last_clean":"2024-03-30T13:03:38.972258+0000","last_became_active":"2024-03-30T13:03:25.175168+0000","last_became_peered":"2024-03-30T13:03:25.175168+0000","last_unstale":"2024-03-30T13:03:38.972258+0000","last_undegraded":"2024-03-30T13:03:38.972258+0000","last_fullsized":"2024-03-30T13:03:38.972258+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:02:33.302956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:33.302956+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":61,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-30T13:03:38.971928+0000","last_change":"2024-03-30T13:03:29.559445+0000","last_active":"2024-03-30T13:03:38.971928+0000","last_peered":"2024-03-30T13:03:38.971928+0000","last_clean":"2024-03-30T13:03:38.971928+0000","last_became_active":"2024-03-30T13:03:25.178075+0000","last_became_peered":"2024-03-30T13:03:25.178075+0000","last_unstale":"2024-03-30T13:03:38.971928+0000","last_undegraded":"2024-03-30T13:03:38.971928+0000","last_fullsized":"2024-03-30T13:03:38.971928+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:03:29.559074+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:03:29.559074+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:50.935813+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":146,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-30T13:03:40.470390+0000","last_change":"2024-03-30T13:02:35.984755+0000","last_active":"2024-03-30T13:03:40.470390+0000","last_peered":"2024-03-30T13:03:40.470390+0000","last_clean":"2024-03-30T13:03:40.470390+0000","last_became_active":"2024-03-30T13:01:41.984313+0000","last_became_peered":"2024-03-30T13:01:41.984313+0000","last_unstale":"2024-03-30T13:03:40.470390+0000","last_undegraded":"2024-03-30T13:03:40.470390+0000","last_fullsized":"2024-03-30T13:03:40.470390+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:35.984506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:35.984506+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:55.717882+0000","scrub_duration":291,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":495,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-03-30T13:03:36.365479+0000","last_change":"2024-03-30T13:02:10.596835+0000","last_active":"2024-03-30T13:03:36.365479+0000","last_peered":"2024-03-30T13:03:36.365479+0000","last_clean":"2024-03-30T13:03:36.365479+0000","last_became_active":"2024-03-30T13:01:41.987920+0000","last_became_peered":"2024-03-30T13:01:41.987920+0000","last_unstale":"2024-03-30T13:03:36.365479+0000","last_undegraded":"2024-03-30T13:03:36.365479+0000","last_fullsized":"2024-03-30T13:03:36.365479+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:02:10.596458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:10.596458+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:38.665640+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22789962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":82,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-30T13:03:38.971958+0000","last_change":"2024-03-30T13:03:25.276567+0000","last_active":"2024-03-30T13:03:38.971958+0000","last_peered":"2024-03-30T13:03:38.971958+0000","last_clean":"2024-03-30T13:03:38.971958+0000","last_became_active":"2024-03-30T13:03:25.178475+0000","last_became_peered":"2024-03-30T13:03:25.178475+0000","last_unstale":"2024-03-30T13:03:38.971958+0000","last_undegraded":"2024-03-30T13:03:38.971958+0000","last_fullsized":"2024-03-30T13:03:38.971958+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:03:13.094988+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:03:13.094988+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:21.173842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":486,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-30T13:03:37.307252+0000","last_change":"2024-03-30T13:02:37.318261+0000","last_active":"2024-03-30T13:03:37.307252+0000","last_peered":"2024-03-30T13:03:37.307252+0000","last_clean":"2024-03-30T13:03:37.307252+0000","last_became_active":"2024-03-30T13:01:41.988151+0000","last_became_peered":"2024-03-30T13:01:41.988151+0000","last_unstale":"2024-03-30T13:03:37.307252+0000","last_undegraded":"2024-03-30T13:03:37.307252+0000","last_fullsized":"2024-03-30T13:03:37.307252+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:02:37.317994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:37.317994+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:02.570664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":22033680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":573,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-03-30T13:03:36.365742+0000","last_change":"2024-03-30T13:03:30.573752+0000","last_active":"2024-03-30T13:03:36.365742+0000","last_peered":"2024-03-30T13:03:36.365742+0000","last_clean":"2024-03-30T13:03:36.365742+0000","last_became_active":"2024-03-30T13:03:25.980366+0000","last_became_peered":"2024-03-30T13:03:25.980366+0000","last_unstale":"2024-03-30T13:03:36.365742+0000","last_undegraded":"2024-03-30T13:03:36.365742+0000","last_fullsized":"2024-03-30T13:03:36.365742+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:03:30.573489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:30.573489+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:39.840213+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":45680832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":379,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-30T13:03:39.326255+0000","last_change":"2024-03-30T13:02:28.264089+0000","last_active":"2024-03-30T13:03:39.326255+0000","last_peered":"2024-03-30T13:03:39.326255+0000","last_clean":"2024-03-30T13:03:39.326255+0000","last_became_active":"2024-03-30T13:00:46.109054+0000","last_became_peered":"2024-03-30T13:00:46.109054+0000","last_unstale":"2024-03-30T13:03:39.326255+0000","last_undegraded":"2024-03-30T13:03:39.326255+0000","last_fullsized":"2024-03-30T13:03:39.326255+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:28.263717+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:02:28.263717+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:56.652431+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":511,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-30T13:03:39.554687+0000","last_change":"2024-03-30T13:02:18.934000+0000","last_active":"2024-03-30T13:03:39.554687+0000","last_peered":"2024-03-30T13:03:39.554687+0000","last_clean":"2024-03-30T13:03:39.554687+0000","last_became_active":"2024-03-30T13:00:46.106211+0000","last_became_peered":"2024-03-30T13:00:46.106211+0000","last_unstale":"2024-03-30T13:03:39.554687+0000","last_undegraded":"2024-03-30T13:03:39.554687+0000","last_fullsized":"2024-03-30T13:03:39.554687+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:02:18.933526+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:42.090625+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25479082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":527,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-30T13:03:39.554759+0000","last_change":"2024-03-30T13:03:28.656036+0000","last_active":"2024-03-30T13:03:39.554759+0000","last_peered":"2024-03-30T13:03:39.554759+0000","last_clean":"2024-03-30T13:03:39.554759+0000","last_became_active":"2024-03-30T13:01:41.991185+0000","last_became_peered":"2024-03-30T13:01:41.991185+0000","last_unstale":"2024-03-30T13:03:39.554759+0000","last_undegraded":"2024-03-30T13:03:39.554759+0000","last_fullsized":"2024-03-30T13:03:39.554759+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:03:28.655753+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:46.701130+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":160,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-30T13:03:40.470527+0000","last_change":"2024-03-30T13:03:01.546561+0000","last_active":"2024-03-30T13:03:40.470527+0000","last_peered":"2024-03-30T13:03:40.470527+0000","last_clean":"2024-03-30T13:03:40.470527+0000","last_became_active":"2024-03-30T13:01:41.982566+0000","last_became_peered":"2024-03-30T13:01:41.982566+0000","last_unstale":"2024-03-30T13:03:40.470527+0000","last_undegraded":"2024-03-30T13:03:40.470527+0000","last_fullsized":"2024-03-30T13:03:40.470527+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:03:01.546292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:01.546292+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:20.359919+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":59,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-30T13:03:38.971928+0000","last_change":"2024-03-30T13:03:32.547779+0000","last_active":"2024-03-30T13:03:38.971928+0000","last_peered":"2024-03-30T13:03:38.971928+0000","last_clean":"2024-03-30T13:03:38.971928+0000","last_became_active":"2024-03-30T13:03:25.178234+0000","last_became_peered":"2024-03-30T13:03:25.178234+0000","last_unstale":"2024-03-30T13:03:38.971928+0000","last_undegraded":"2024-03-30T13:03:38.971928+0000","last_fullsized":"2024-03-30T13:03:38.971928+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:03:32.547475+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:03:32.547475+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:48.688284+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":160,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-30T13:03:40.470839+0000","last_change":"2024-03-30T13:02:56.617180+0000","last_active":"2024-03-30T13:03:40.470839+0000","last_peered":"2024-03-30T13:03:40.470839+0000","last_clean":"2024-03-30T13:03:40.470839+0000","last_became_active":"2024-03-30T13:01:41.983624+0000","last_became_peered":"2024-03-30T13:01:41.983624+0000","last_unstale":"2024-03-30T13:03:40.470839+0000","last_undegraded":"2024-03-30T13:03:40.470839+0000","last_fullsized":"2024-03-30T13:03:40.470839+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:02:56.616926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:56.616926+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:25.091190+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":472,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-30T13:03:37.307263+0000","last_change":"2024-03-30T13:02:40.333877+0000","last_active":"2024-03-30T13:03:37.307263+0000","last_peered":"2024-03-30T13:03:37.307263+0000","last_clean":"2024-03-30T13:03:37.307263+0000","last_became_active":"2024-03-30T13:01:41.986001+0000","last_became_peered":"2024-03-30T13:01:41.986001+0000","last_unstale":"2024-03-30T13:03:37.307263+0000","last_undegraded":"2024-03-30T13:03:37.307263+0000","last_fullsized":"2024-03-30T13:03:37.307263+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:02:40.333589+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:02:40.333589+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:45.470373+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10453822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":233513290,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509541,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48868,"kb_used_data":17688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322972,"statfs":{"total":96636764160,"available":96586723328,"internally_reserved":0,"allocated":18112512,"data_stored":16646391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":156,"seq":670014898203,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54156,"kb_used_data":22456,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94317684,"statfs":{"total":96636764160,"available":96581308416,"internally_reserved":0,"allocated":22994944,"data_stored":21515221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4405,"internal_metadata":32435915},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":135,"seq":579820584997,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46304,"kb_used_data":14160,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94325536,"statfs":{"total":96636764160,"available":96589348864,"internally_reserved":0,"allocated":14499840,"data_stored":13020117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6047,"internal_metadata":32893025},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509541,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40092,"kb_used_data":8904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331748,"statfs":{"total":96636764160,"available":96595709952,"internally_reserved":0,"allocated":9117696,"data_stored":7651575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285626,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39436,"kb_used_data":7736,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94332404,"statfs":{"total":96636764160,"available":96596381696,"internally_reserved":0,"allocated":7921664,"data_stored":6462344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509541,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47276,"kb_used_data":16092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324564,"statfs":{"total":96636764160,"available":96588353536,"internally_reserved":0,"allocated":16478208,"data_stored":15005286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":206,"seq":884763262981,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45656,"kb_used_data":12612,"kb_used_omap":9,"kb_used_meta":33014,"kb_avail":94326184,"statfs":{"total":96636764160,"available":96590012416,"internally_reserved":0,"allocated":12914688,"data_stored":11455368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9338,"internal_metadata":33807238},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574949,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42096,"kb_used_data":15068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329744,"statfs":{"total":96636764160,"available":96593657856,"internally_reserved":0,"allocated":15429632,"data_stored":13970312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:03:42.508 INFO:tasks.thrashosds.thrasher:Choosing PG 1.f with acting set [6, 5, 4] 2024-03-30T13:03:42.509 INFO:tasks.thrashosds.thrasher:Killing osd 6, live_osds are [0, 2, 3, 6, 1, 4, 5, 7] 2024-03-30T13:03:42.509 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-30T13:03:42.509 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:03:42.591 INFO:tasks.ceph.osd.6:Stopped 2024-03-30T13:03:42.591 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 2, 3, 6, 1, 4, 5, 7] 2024-03-30T13:03:42.591 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 6 2024-03-30T13:03:42.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:42.711+0000 7f396acd3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:42.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:42.723+0000 7f396acd3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:42.914 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:03:42.911+0000 7fbb38f08640 -1 mon.a@0(leader).osd e222 definitely_dead 0 2024-03-30T13:03:43.359 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.6. 2024-03-30T13:03:43.392 INFO:tasks.thrashosds.thrasher:dead_osds=[6], live_osds=[0, 2, 3, 1, 4, 5, 7] 2024-03-30T13:03:43.392 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:03:43.393 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:03:43.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:43.503+0000 7f578f0e5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:43.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:03:43.515+0000 7f578f0e5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:03:43.687 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:03:43.688 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:03:43.703 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":461,"stamp":"2024-03-30T13:03:43.353530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":233513290,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":41,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":315096,"kb_used_data":97124,"kb_used_omap":28,"kb_used_meta":217827,"kb_avail":660287784,"statfs":{"total":676457349120,"available":676134690816,"internally_reserved":0,"allocated":99454976,"data_stored":89124119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28975,"internal_metadata":223055569},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.432715"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":507,"reported_epoch":221,"state":"stale+active+clean","last_fresh":"2024-03-30T13:03:39.554648+0000","last_change":"2024-03-30T13:02:52.868231+0000","last_active":"2024-03-30T13:03:39.554648+0000","last_peered":"2024-03-30T13:03:39.554648+0000","last_clean":"2024-03-30T13:03:39.554648+0000","last_became_active":"2024-03-30T13:01:41.987359+0000","last_became_peered":"2024-03-30T13:01:41.987359+0000","last_unstale":"2024-03-30T13:03:43.353563+0000","last_undegraded":"2024-03-30T13:03:39.554648+0000","last_fullsized":"2024-03-30T13:03:39.554648+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:02:52.867981+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:22.928753+0000","last_clean_scrub_stamp":"2024-03-30T13:02:52.867981+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:09.169753+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18352980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":478,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-30T13:03:39.326382+0000","last_change":"2024-03-30T13:03:25.182999+0000","last_active":"2024-03-30T13:03:39.326382+0000","last_peered":"2024-03-30T13:03:39.326382+0000","last_clean":"2024-03-30T13:03:39.326382+0000","last_became_active":"2024-03-30T13:03:25.181879+0000","last_became_peered":"2024-03-30T13:03:25.181879+0000","last_unstale":"2024-03-30T13:03:39.326382+0000","last_undegraded":"2024-03-30T13:03:39.326382+0000","last_fullsized":"2024-03-30T13:03:39.326382+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:27.269301+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:02:27.269301+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:43.734307+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10621892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":55,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-30T13:03:38.972258+0000","last_change":"2024-03-30T13:03:25.176251+0000","last_active":"2024-03-30T13:03:38.972258+0000","last_peered":"2024-03-30T13:03:38.972258+0000","last_clean":"2024-03-30T13:03:38.972258+0000","last_became_active":"2024-03-30T13:03:25.175168+0000","last_became_peered":"2024-03-30T13:03:25.175168+0000","last_unstale":"2024-03-30T13:03:38.972258+0000","last_undegraded":"2024-03-30T13:03:38.972258+0000","last_fullsized":"2024-03-30T13:03:38.972258+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:02:33.302956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:33.302956+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":61,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-30T13:03:38.971928+0000","last_change":"2024-03-30T13:03:29.559445+0000","last_active":"2024-03-30T13:03:38.971928+0000","last_peered":"2024-03-30T13:03:38.971928+0000","last_clean":"2024-03-30T13:03:38.971928+0000","last_became_active":"2024-03-30T13:03:25.178075+0000","last_became_peered":"2024-03-30T13:03:25.178075+0000","last_unstale":"2024-03-30T13:03:38.971928+0000","last_undegraded":"2024-03-30T13:03:38.971928+0000","last_fullsized":"2024-03-30T13:03:38.971928+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:03:29.559074+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:03:29.559074+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:50.935813+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":146,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-30T13:03:40.470390+0000","last_change":"2024-03-30T13:02:35.984755+0000","last_active":"2024-03-30T13:03:40.470390+0000","last_peered":"2024-03-30T13:03:40.470390+0000","last_clean":"2024-03-30T13:03:40.470390+0000","last_became_active":"2024-03-30T13:01:41.984313+0000","last_became_peered":"2024-03-30T13:01:41.984313+0000","last_unstale":"2024-03-30T13:03:40.470390+0000","last_undegraded":"2024-03-30T13:03:40.470390+0000","last_fullsized":"2024-03-30T13:03:40.470390+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:35.984506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:35.984506+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:55.717882+0000","scrub_duration":291,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":502,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-30T13:03:40.470378+0000","last_change":"2024-03-30T13:02:10.596835+0000","last_active":"2024-03-30T13:03:40.470378+0000","last_peered":"2024-03-30T13:03:40.470378+0000","last_clean":"2024-03-30T13:03:40.470378+0000","last_became_active":"2024-03-30T13:01:41.987920+0000","last_became_peered":"2024-03-30T13:01:41.987920+0000","last_unstale":"2024-03-30T13:03:40.470378+0000","last_undegraded":"2024-03-30T13:03:40.470378+0000","last_fullsized":"2024-03-30T13:03:40.470378+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:02:10.596458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:10.596458+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22789962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":82,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-30T13:03:38.971958+0000","last_change":"2024-03-30T13:03:25.276567+0000","last_active":"2024-03-30T13:03:38.971958+0000","last_peered":"2024-03-30T13:03:38.971958+0000","last_clean":"2024-03-30T13:03:38.971958+0000","last_became_active":"2024-03-30T13:03:25.178475+0000","last_became_peered":"2024-03-30T13:03:25.178475+0000","last_unstale":"2024-03-30T13:03:38.971958+0000","last_undegraded":"2024-03-30T13:03:38.971958+0000","last_fullsized":"2024-03-30T13:03:38.971958+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:03:13.094988+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:03:13.094988+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:21.173842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":492,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-30T13:03:40.347146+0000","last_change":"2024-03-30T13:02:37.318261+0000","last_active":"2024-03-30T13:03:40.347146+0000","last_peered":"2024-03-30T13:03:40.347146+0000","last_clean":"2024-03-30T13:03:40.347146+0000","last_became_active":"2024-03-30T13:01:41.988151+0000","last_became_peered":"2024-03-30T13:01:41.988151+0000","last_unstale":"2024-03-30T13:03:40.347146+0000","last_undegraded":"2024-03-30T13:03:40.347146+0000","last_fullsized":"2024-03-30T13:03:40.347146+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:02:37.317994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:37.317994+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:02.570664+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":22033680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":580,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-30T13:03:40.470499+0000","last_change":"2024-03-30T13:03:30.573752+0000","last_active":"2024-03-30T13:03:40.470499+0000","last_peered":"2024-03-30T13:03:40.470499+0000","last_clean":"2024-03-30T13:03:40.470499+0000","last_became_active":"2024-03-30T13:03:25.980366+0000","last_became_peered":"2024-03-30T13:03:25.980366+0000","last_unstale":"2024-03-30T13:03:40.470499+0000","last_undegraded":"2024-03-30T13:03:40.470499+0000","last_fullsized":"2024-03-30T13:03:40.470499+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:03:30.573489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:30.573489+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:39.840213+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":45680832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":379,"reported_epoch":221,"state":"active+clean","last_fresh":"2024-03-30T13:03:39.326255+0000","last_change":"2024-03-30T13:02:28.264089+0000","last_active":"2024-03-30T13:03:39.326255+0000","last_peered":"2024-03-30T13:03:39.326255+0000","last_clean":"2024-03-30T13:03:39.326255+0000","last_became_active":"2024-03-30T13:00:46.109054+0000","last_became_peered":"2024-03-30T13:00:46.109054+0000","last_unstale":"2024-03-30T13:03:39.326255+0000","last_undegraded":"2024-03-30T13:03:39.326255+0000","last_fullsized":"2024-03-30T13:03:39.326255+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:02:28.263717+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:02:28.263717+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:56.652431+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":511,"reported_epoch":221,"state":"stale+active+clean","last_fresh":"2024-03-30T13:03:39.554687+0000","last_change":"2024-03-30T13:02:18.934000+0000","last_active":"2024-03-30T13:03:39.554687+0000","last_peered":"2024-03-30T13:03:39.554687+0000","last_clean":"2024-03-30T13:03:39.554687+0000","last_became_active":"2024-03-30T13:00:46.106211+0000","last_became_peered":"2024-03-30T13:00:46.106211+0000","last_unstale":"2024-03-30T13:03:43.353570+0000","last_undegraded":"2024-03-30T13:03:39.554687+0000","last_fullsized":"2024-03-30T13:03:39.554687+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:02:18.933526+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:42.090625+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":4580,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25479082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":527,"reported_epoch":221,"state":"stale+active+clean","last_fresh":"2024-03-30T13:03:39.554759+0000","last_change":"2024-03-30T13:03:28.656036+0000","last_active":"2024-03-30T13:03:39.554759+0000","last_peered":"2024-03-30T13:03:39.554759+0000","last_clean":"2024-03-30T13:03:39.554759+0000","last_became_active":"2024-03-30T13:01:41.991185+0000","last_became_peered":"2024-03-30T13:01:41.991185+0000","last_unstale":"2024-03-30T13:03:43.353574+0000","last_undegraded":"2024-03-30T13:03:39.554759+0000","last_fullsized":"2024-03-30T13:03:39.554759+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:03:28.655753+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:46.701130+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":160,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-30T13:03:40.470527+0000","last_change":"2024-03-30T13:03:01.546561+0000","last_active":"2024-03-30T13:03:40.470527+0000","last_peered":"2024-03-30T13:03:40.470527+0000","last_clean":"2024-03-30T13:03:40.470527+0000","last_became_active":"2024-03-30T13:01:41.982566+0000","last_became_peered":"2024-03-30T13:01:41.982566+0000","last_unstale":"2024-03-30T13:03:40.470527+0000","last_undegraded":"2024-03-30T13:03:40.470527+0000","last_fullsized":"2024-03-30T13:03:40.470527+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:03:01.546292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:01.546292+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:20.359919+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":59,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-03-30T13:03:38.971928+0000","last_change":"2024-03-30T13:03:32.547779+0000","last_active":"2024-03-30T13:03:38.971928+0000","last_peered":"2024-03-30T13:03:38.971928+0000","last_clean":"2024-03-30T13:03:38.971928+0000","last_became_active":"2024-03-30T13:03:25.178234+0000","last_became_peered":"2024-03-30T13:03:25.178234+0000","last_unstale":"2024-03-30T13:03:38.971928+0000","last_undegraded":"2024-03-30T13:03:38.971928+0000","last_fullsized":"2024-03-30T13:03:38.971928+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:03:32.547475+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:03:32.547475+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:48.688284+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":160,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-30T13:03:40.470839+0000","last_change":"2024-03-30T13:02:56.617180+0000","last_active":"2024-03-30T13:03:40.470839+0000","last_peered":"2024-03-30T13:03:40.470839+0000","last_clean":"2024-03-30T13:03:40.470839+0000","last_became_active":"2024-03-30T13:01:41.983624+0000","last_became_peered":"2024-03-30T13:01:41.983624+0000","last_unstale":"2024-03-30T13:03:40.470839+0000","last_undegraded":"2024-03-30T13:03:40.470839+0000","last_fullsized":"2024-03-30T13:03:40.470839+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:02:56.616926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:56.616926+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:25.091190+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":478,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-03-30T13:03:40.347157+0000","last_change":"2024-03-30T13:02:40.333877+0000","last_active":"2024-03-30T13:03:40.347157+0000","last_peered":"2024-03-30T13:03:40.347157+0000","last_clean":"2024-03-30T13:03:40.347157+0000","last_became_active":"2024-03-30T13:01:41.986001+0000","last_became_peered":"2024-03-30T13:01:41.986001+0000","last_unstale":"2024-03-30T13:03:40.347157+0000","last_undegraded":"2024-03-30T13:03:40.347157+0000","last_fullsized":"2024-03-30T13:03:40.347157+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:02:40.333589+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:02:40.333589+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:03:45.470373+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10453822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":91,"num_read_kb":4612,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":233513290,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":156,"seq":670014898203,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54156,"kb_used_data":22456,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94317684,"statfs":{"total":96636764160,"available":96581308416,"internally_reserved":0,"allocated":22994944,"data_stored":21515221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4405,"internal_metadata":32435915},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":135,"seq":579820584997,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46304,"kb_used_data":14160,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94325536,"statfs":{"total":96636764160,"available":96589348864,"internally_reserved":0,"allocated":14499840,"data_stored":13020117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6047,"internal_metadata":32893025},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509541,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40092,"kb_used_data":8904,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331748,"statfs":{"total":96636764160,"available":96595709952,"internally_reserved":0,"allocated":9117696,"data_stored":7651575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285627,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39476,"kb_used_data":7784,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94332364,"statfs":{"total":96636764160,"available":96596340736,"internally_reserved":0,"allocated":7970816,"data_stored":6484292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509541,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47276,"kb_used_data":16092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324564,"statfs":{"total":96636764160,"available":96588353536,"internally_reserved":0,"allocated":16478208,"data_stored":15005286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":206,"seq":884763262981,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45656,"kb_used_data":12612,"kb_used_omap":9,"kb_used_meta":33014,"kb_avail":94326184,"statfs":{"total":96636764160,"available":96590012416,"internally_reserved":0,"allocated":12914688,"data_stored":11455368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9338,"internal_metadata":33807238},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574950,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42136,"kb_used_data":15116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329704,"statfs":{"total":96636764160,"available":96593616896,"internally_reserved":0,"allocated":15478784,"data_stored":13992260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:03:43.704 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:04:08.705 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:04:08.705 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:04:08.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:08.831+0000 7fa427989640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:08.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:08.843+0000 7fa427989640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:09.020 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:04:09.021 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:04:09.037 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":476,"stamp":"2024-03-30T13:04:07.929335+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":211109856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":331772,"kb_used_data":113832,"kb_used_omap":28,"kb_used_meta":217827,"kb_avail":660271108,"statfs":{"total":676457349120,"available":676117614592,"internally_reserved":0,"allocated":116563968,"data_stored":106017432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28975,"internal_metadata":223055569},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002785"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":19,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-30T13:04:01.380820+0000","last_change":"2024-03-30T13:03:45.492361+0000","last_active":"2024-03-30T13:04:01.380820+0000","last_peered":"2024-03-30T13:04:01.380820+0000","last_clean":"2024-03-30T13:04:01.380820+0000","last_became_active":"2024-03-30T13:03:44.369563+0000","last_became_peered":"2024-03-30T13:03:44.369563+0000","last_unstale":"2024-03-30T13:04:01.380820+0000","last_undegraded":"2024-03-30T13:04:01.380820+0000","last_fullsized":"2024-03-30T13:04:01.380820+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:02:52.867981+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:22.928753+0000","last_clean_scrub_stamp":"2024-03-30T13:02:52.867981+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:21.649239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":494,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-30T13:03:59.011746+0000","last_change":"2024-03-30T13:03:51.058827+0000","last_active":"2024-03-30T13:03:59.011746+0000","last_peered":"2024-03-30T13:03:59.011746+0000","last_clean":"2024-03-30T13:03:59.011746+0000","last_became_active":"2024-03-30T13:03:25.181879+0000","last_became_peered":"2024-03-30T13:03:25.181879+0000","last_unstale":"2024-03-30T13:03:59.011746+0000","last_undegraded":"2024-03-30T13:03:59.011746+0000","last_fullsized":"2024-03-30T13:03:59.011746+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:03:51.058584+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:03:51.058584+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:02.988983+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10621892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":74,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-30T13:04:00.588021+0000","last_change":"2024-03-30T13:03:39.460832+0000","last_active":"2024-03-30T13:04:00.588021+0000","last_peered":"2024-03-30T13:04:00.588021+0000","last_clean":"2024-03-30T13:04:00.588021+0000","last_became_active":"2024-03-30T13:03:25.175168+0000","last_became_peered":"2024-03-30T13:03:25.175168+0000","last_unstale":"2024-03-30T13:04:00.588021+0000","last_undegraded":"2024-03-30T13:04:00.588021+0000","last_fullsized":"2024-03-30T13:04:00.588021+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_clean_scrub_stamp":"2024-03-30T13:03:39.460563+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:54.437907+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":84,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-30T13:04:00.587760+0000","last_change":"2024-03-30T13:03:44.420524+0000","last_active":"2024-03-30T13:04:00.587760+0000","last_peered":"2024-03-30T13:04:00.587760+0000","last_clean":"2024-03-30T13:04:00.587760+0000","last_became_active":"2024-03-30T13:03:44.372643+0000","last_became_peered":"2024-03-30T13:03:44.372643+0000","last_unstale":"2024-03-30T13:04:00.587760+0000","last_undegraded":"2024-03-30T13:04:00.587760+0000","last_fullsized":"2024-03-30T13:04:00.587760+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:03:29.559074+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:03:29.559074+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:52.643084+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":161,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-30T13:04:00.375103+0000","last_change":"2024-03-30T13:03:58.424647+0000","last_active":"2024-03-30T13:04:00.375103+0000","last_peered":"2024-03-30T13:04:00.375103+0000","last_clean":"2024-03-30T13:04:00.375103+0000","last_became_active":"2024-03-30T13:01:41.984313+0000","last_became_peered":"2024-03-30T13:01:41.984313+0000","last_unstale":"2024-03-30T13:04:00.375103+0000","last_undegraded":"2024-03-30T13:04:00.375103+0000","last_fullsized":"2024-03-30T13:04:00.375103+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:03:58.424408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:58.424408+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:16.577293+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":516,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-30T13:04:01.380913+0000","last_change":"2024-03-30T13:03:49.426733+0000","last_active":"2024-03-30T13:04:01.380913+0000","last_peered":"2024-03-30T13:04:01.380913+0000","last_clean":"2024-03-30T13:04:01.380913+0000","last_became_active":"2024-03-30T13:01:41.987920+0000","last_became_peered":"2024-03-30T13:01:41.987920+0000","last_unstale":"2024-03-30T13:04:01.380913+0000","last_undegraded":"2024-03-30T13:04:01.380913+0000","last_fullsized":"2024-03-30T13:04:01.380913+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:03:49.426454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:49.426454+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:10.035429+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22789962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":93,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-30T13:04:00.587972+0000","last_change":"2024-03-30T13:03:25.276567+0000","last_active":"2024-03-30T13:04:00.587972+0000","last_peered":"2024-03-30T13:04:00.587972+0000","last_clean":"2024-03-30T13:04:00.587972+0000","last_became_active":"2024-03-30T13:03:25.178475+0000","last_became_peered":"2024-03-30T13:03:25.178475+0000","last_unstale":"2024-03-30T13:04:00.587972+0000","last_undegraded":"2024-03-30T13:04:00.587972+0000","last_fullsized":"2024-03-30T13:04:00.587972+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:03:13.094988+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:03:13.094988+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:21.173842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":520,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-30T13:04:01.041008+0000","last_change":"2024-03-30T13:03:57.023935+0000","last_active":"2024-03-30T13:04:01.041008+0000","last_peered":"2024-03-30T13:04:01.041008+0000","last_clean":"2024-03-30T13:04:01.041008+0000","last_became_active":"2024-03-30T13:03:44.376344+0000","last_became_peered":"2024-03-30T13:03:44.376344+0000","last_unstale":"2024-03-30T13:04:01.041008+0000","last_undegraded":"2024-03-30T13:04:01.041008+0000","last_fullsized":"2024-03-30T13:04:01.041008+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:03:57.023686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:57.023686+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:15.235590+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29378240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":587,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-30T13:04:01.381152+0000","last_change":"2024-03-30T13:03:30.573752+0000","last_active":"2024-03-30T13:04:01.381152+0000","last_peered":"2024-03-30T13:04:01.381152+0000","last_clean":"2024-03-30T13:04:01.381152+0000","last_became_active":"2024-03-30T13:03:25.980366+0000","last_became_peered":"2024-03-30T13:03:25.980366+0000","last_unstale":"2024-03-30T13:04:01.381152+0000","last_undegraded":"2024-03-30T13:04:01.381152+0000","last_fullsized":"2024-03-30T13:04:01.381152+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:03:30.573489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:30.573489+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:39.840213+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":45680832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":405,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-30T13:03:59.011677+0000","last_change":"2024-03-30T13:03:44.984800+0000","last_active":"2024-03-30T13:03:59.011677+0000","last_peered":"2024-03-30T13:03:59.011677+0000","last_clean":"2024-03-30T13:03:59.011677+0000","last_became_active":"2024-03-30T13:03:44.373958+0000","last_became_peered":"2024-03-30T13:03:44.373958+0000","last_unstale":"2024-03-30T13:03:59.011677+0000","last_undegraded":"2024-03-30T13:03:59.011677+0000","last_fullsized":"2024-03-30T13:03:59.011677+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:03:44.984484+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:03:44.984484+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:46.349906+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":33,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-30T13:04:00.587784+0000","last_change":"2024-03-30T13:03:46.561082+0000","last_active":"2024-03-30T13:04:00.587784+0000","last_peered":"2024-03-30T13:04:00.587784+0000","last_clean":"2024-03-30T13:04:00.587784+0000","last_became_active":"2024-03-30T13:03:44.372551+0000","last_became_peered":"2024-03-30T13:03:44.372551+0000","last_unstale":"2024-03-30T13:04:00.587784+0000","last_undegraded":"2024-03-30T13:04:00.587784+0000","last_fullsized":"2024-03-30T13:04:00.587784+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:03:46.560815+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:03:46.560815+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:13.803655+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":31,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-30T13:04:00.588017+0000","last_change":"2024-03-30T13:03:45.645627+0000","last_active":"2024-03-30T13:04:00.588017+0000","last_peered":"2024-03-30T13:04:00.588017+0000","last_clean":"2024-03-30T13:04:00.588017+0000","last_became_active":"2024-03-30T13:03:44.368109+0000","last_became_peered":"2024-03-30T13:03:44.368109+0000","last_unstale":"2024-03-30T13:04:00.588017+0000","last_undegraded":"2024-03-30T13:04:00.588017+0000","last_fullsized":"2024-03-30T13:04:00.588017+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:03:28.655753+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:49.997729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":167,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-30T13:04:00.375077+0000","last_change":"2024-03-30T13:03:01.546561+0000","last_active":"2024-03-30T13:04:00.375077+0000","last_peered":"2024-03-30T13:04:00.375077+0000","last_clean":"2024-03-30T13:04:00.375077+0000","last_became_active":"2024-03-30T13:01:41.982566+0000","last_became_peered":"2024-03-30T13:01:41.982566+0000","last_unstale":"2024-03-30T13:04:00.375077+0000","last_undegraded":"2024-03-30T13:04:00.375077+0000","last_fullsized":"2024-03-30T13:04:00.375077+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:03:01.546292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:01.546292+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:20.359919+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":70,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-30T13:04:00.587888+0000","last_change":"2024-03-30T13:03:32.547779+0000","last_active":"2024-03-30T13:04:00.587888+0000","last_peered":"2024-03-30T13:04:00.587888+0000","last_clean":"2024-03-30T13:04:00.587888+0000","last_became_active":"2024-03-30T13:03:25.178234+0000","last_became_peered":"2024-03-30T13:03:25.178234+0000","last_unstale":"2024-03-30T13:04:00.587888+0000","last_undegraded":"2024-03-30T13:04:00.587888+0000","last_fullsized":"2024-03-30T13:04:00.587888+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:03:32.547475+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:03:32.547475+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:48.688284+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":183,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-30T13:04:00.375066+0000","last_change":"2024-03-30T13:03:44.450573+0000","last_active":"2024-03-30T13:04:00.375066+0000","last_peered":"2024-03-30T13:04:00.375066+0000","last_clean":"2024-03-30T13:04:00.375066+0000","last_became_active":"2024-03-30T13:03:44.373559+0000","last_became_peered":"2024-03-30T13:03:44.373559+0000","last_unstale":"2024-03-30T13:04:00.375066+0000","last_undegraded":"2024-03-30T13:04:00.375066+0000","last_fullsized":"2024-03-30T13:04:00.375066+0000","mapping_epoch":223,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":224,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:02:56.616926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:02:56.616926+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:08.944900+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":492,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-03-30T13:04:01.041346+0000","last_change":"2024-03-30T13:03:49.092146+0000","last_active":"2024-03-30T13:04:01.041346+0000","last_peered":"2024-03-30T13:04:01.041346+0000","last_clean":"2024-03-30T13:04:01.041346+0000","last_became_active":"2024-03-30T13:01:41.986001+0000","last_became_peered":"2024-03-30T13:01:41.986001+0000","last_unstale":"2024-03-30T13:04:01.041346+0000","last_undegraded":"2024-03-30T13:04:01.041346+0000","last_fullsized":"2024-03-30T13:04:01.041346+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:03:49.091864+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:03:49.091864+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:51.096108+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10453822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":211109856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":113025024,"data_stored":113025024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":156,"seq":670014898208,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54196,"kb_used_data":22504,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94317644,"statfs":{"total":96636764160,"available":96581267456,"internally_reserved":0,"allocated":23044096,"data_stored":21537448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4405,"internal_metadata":32435915},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":135,"seq":579820585002,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46348,"kb_used_data":14208,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94325492,"statfs":{"total":96636764160,"available":96589303808,"internally_reserved":0,"allocated":14548992,"data_stored":13042344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6047,"internal_metadata":32893025},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509546,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41472,"kb_used_data":10292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330368,"statfs":{"total":96636764160,"available":96594296832,"internally_reserved":0,"allocated":10539008,"data_stored":9032360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285631,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39512,"kb_used_data":7820,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94332328,"statfs":{"total":96636764160,"available":96596303872,"internally_reserved":0,"allocated":8007680,"data_stored":6501032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509546,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52432,"kb_used_data":21252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94319408,"statfs":{"total":96636764160,"available":96583073792,"internally_reserved":0,"allocated":21762048,"data_stored":20255400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":206,"seq":884763262986,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54612,"kb_used_data":21576,"kb_used_omap":9,"kb_used_meta":33014,"kb_avail":94317228,"statfs":{"total":96636764160,"available":96580841472,"internally_reserved":0,"allocated":22093824,"data_stored":20587176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9338,"internal_metadata":33807238},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574955,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43200,"kb_used_data":16180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94328640,"statfs":{"total":96636764160,"available":96592527360,"internally_reserved":0,"allocated":16568320,"data_stored":15061672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19046400,"data_stored":19046400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7823360,"data_stored":7823360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":13852672,"data_stored":13852672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":19378176,"data_stored":19378176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:04:09.038 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:04:09.038 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:04:09.039 INFO:tasks.thrashosds.thrasher:Reviving osd 6 2024-03-30T13:04:09.039 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-30T13:04:09.039 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-30T13:04:09.042 INFO:tasks.ceph.osd.6:Started 2024-03-30T13:04:09.042 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-30T13:04:09.109 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T13:04:09.106+0000 7f16f7cbe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:09.117 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T13:04:09.118+0000 7f16f7cbe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:09.118 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T13:04:09.118+0000 7f16f7cbe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:09.165 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T13:04:09.165 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T13:04:09.165 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T13:04:09.165 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T13:04:09.166 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T13:04:09.166 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T13:04:09.166 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-03-30T13:04:09.166 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-03-30T13:04:09.166 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-03-30T13:04:09.166 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-03-30T13:04:09.166 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-03-30T13:04:09.166 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T13:04:09.167 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:04:09.168 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-03-30T13:04:09.651 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T13:04:09.650+0000 7f16f7cbe7c0 -1 Falling back to public interface 2024-03-30T13:04:13.476 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T13:04:13.474+0000 7f16f7cbe7c0 -1 osd.6 222 log_to_monitors true 2024-03-30T13:04:14.046 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T13:04:14.046+0000 7f16eba8b640 -1 osd.6 222 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:04:14.169 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-30T13:04:14.301 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T13:04:14.301 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T13:04:14.301 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T13:04:14.301 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T13:04:14.315 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:04:14.315 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-03-30T13:04:14.315 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 6 2024-03-30T13:04:14.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:14.439+0000 7f9fb14e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:14.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:14.455+0000 7f9fb14e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:14.637 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:04:14.635+0000 7fbb38f08640 -1 mon.a@0(leader).osd e226 definitely_dead 0 2024-03-30T13:04:15.024 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.6. 2024-03-30T13:04:15.053 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-03-30T13:04:15.053 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:04:15.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:15.163+0000 7f71622c0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:15.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:15.183+0000 7f71622c0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:16.028 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:04:16.061 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:04:16.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:16.167+0000 7f1ce1f68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:16.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:16.183+0000 7f1ce1f68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:17.044 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:04:17.067 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:04:17.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:17.179+0000 7f99451c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:17.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:17.195+0000 7f99451c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:18.056 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:04:18.082 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:04:18.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:18.199+0000 7efd1df7c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:18.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:18.215+0000 7efd1df7c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:19.060 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:04:19.091 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:04:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:19.207+0000 7f094e607640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:19.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:19.223+0000 7f094e607640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:20.083 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:04:20.108 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:04:20.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:20.227+0000 7f34838a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:20.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:20.243+0000 7f34838a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:21.085 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:04:21.111 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:04:21.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:21.231+0000 7f157cb49640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:21.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:21.247+0000 7f157cb49640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:22.095 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:04:22.119 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:04:22.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:22.235+0000 7f864c339640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:22.251+0000 7f864c339640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:23.112 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:04:23.137 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:04:23.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:23.255+0000 7f0a82297640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:23.271+0000 7f0a82297640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:24.108 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:04:24.138 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:04:24.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:24.255+0000 7ff794d79640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:24.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:24.275+0000 7ff794d79640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:25.125 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:04:25.149 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:04:25.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:25.259+0000 7f036a79d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:25.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:25.279+0000 7f036a79d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:26.129 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:04:26.154 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:04:26.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:26.263+0000 7f9a0c922640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:26.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:26.283+0000 7f9a0c922640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:27.150 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:04:27.174 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:04:27.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:27.291+0000 7ffbdbc33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:27.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:27.307+0000 7ffbdbc33640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:28.154 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:04:28.179 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:04:28.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:28.295+0000 7faff8d25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:28.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:28.315+0000 7faff8d25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:29.179 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:04:29.205 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:04:29.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:29.327+0000 7fdbb5a8a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:29.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:29.343+0000 7fdbb5a8a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:30.189 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:04:30.211 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:04:30.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:30.323+0000 7f55307ea640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:30.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:30.339+0000 7f55307ea640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:31.183 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:04:31.208 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:04:31.208 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:04:31.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:31.319+0000 7fd910439640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:31.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:31.335+0000 7fd910439640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:31.505 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:04:31.506 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:04:31.522 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2024-03-30T13:04:31.186006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":179126564,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":366304,"kb_used_data":116648,"kb_used_omap":32,"kb_used_meta":249503,"kb_avail":754608416,"statfs":{"total":773094113280,"available":772719017984,"internally_reserved":0,"allocated":119447552,"data_stored":106626421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33373,"internal_metadata":255491491},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.031922"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":508,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-30T13:04:29.981035+0000","last_change":"2024-03-30T13:04:16.526573+0000","last_active":"2024-03-30T13:04:29.981035+0000","last_peered":"2024-03-30T13:04:29.981035+0000","last_clean":"2024-03-30T13:04:29.981035+0000","last_became_active":"2024-03-30T13:04:16.044630+0000","last_became_peered":"2024-03-30T13:04:16.044630+0000","last_unstale":"2024-03-30T13:04:29.981035+0000","last_undegraded":"2024-03-30T13:04:29.981035+0000","last_fullsized":"2024-03-30T13:04:29.981035+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:04:16.526295+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:30.116419+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":524,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-30T13:04:29.176735+0000","last_change":"2024-03-30T13:03:51.058827+0000","last_active":"2024-03-30T13:04:29.176735+0000","last_peered":"2024-03-30T13:04:29.176735+0000","last_clean":"2024-03-30T13:04:29.176735+0000","last_became_active":"2024-03-30T13:03:25.181879+0000","last_became_peered":"2024-03-30T13:03:25.181879+0000","last_unstale":"2024-03-30T13:04:29.176735+0000","last_undegraded":"2024-03-30T13:04:29.176735+0000","last_fullsized":"2024-03-30T13:04:29.176735+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:03:51.058584+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:03:51.058584+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:02.988983+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10621892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":100,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-30T13:04:27.395405+0000","last_change":"2024-03-30T13:03:39.460832+0000","last_active":"2024-03-30T13:04:27.395405+0000","last_peered":"2024-03-30T13:04:27.395405+0000","last_clean":"2024-03-30T13:04:27.395405+0000","last_became_active":"2024-03-30T13:03:25.175168+0000","last_became_peered":"2024-03-30T13:03:25.175168+0000","last_unstale":"2024-03-30T13:04:27.395405+0000","last_undegraded":"2024-03-30T13:04:27.395405+0000","last_fullsized":"2024-03-30T13:04:27.395405+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_clean_scrub_stamp":"2024-03-30T13:03:39.460563+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:54.437907+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":116,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-30T13:04:27.395070+0000","last_change":"2024-03-30T13:04:16.040508+0000","last_active":"2024-03-30T13:04:27.395070+0000","last_peered":"2024-03-30T13:04:27.395070+0000","last_clean":"2024-03-30T13:04:27.395070+0000","last_became_active":"2024-03-30T13:04:16.039346+0000","last_became_peered":"2024-03-30T13:04:16.039346+0000","last_unstale":"2024-03-30T13:04:27.395070+0000","last_undegraded":"2024-03-30T13:04:27.395070+0000","last_fullsized":"2024-03-30T13:04:27.395070+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:03:29.559074+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:03:29.559074+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:34.625794+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":190,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-30T13:04:29.980187+0000","last_change":"2024-03-30T13:03:58.424647+0000","last_active":"2024-03-30T13:04:29.980187+0000","last_peered":"2024-03-30T13:04:29.980187+0000","last_clean":"2024-03-30T13:04:29.980187+0000","last_became_active":"2024-03-30T13:01:41.984313+0000","last_became_peered":"2024-03-30T13:01:41.984313+0000","last_unstale":"2024-03-30T13:04:29.980187+0000","last_undegraded":"2024-03-30T13:04:29.980187+0000","last_fullsized":"2024-03-30T13:04:29.980187+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:03:58.424408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:58.424408+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:16.577293+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":541,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:04:26.135053+0000","last_change":"2024-03-30T13:03:49.426733+0000","last_active":"2024-03-30T13:04:26.135053+0000","last_peered":"2024-03-30T13:04:26.135053+0000","last_clean":"2024-03-30T13:04:26.135053+0000","last_became_active":"2024-03-30T13:01:41.987920+0000","last_became_peered":"2024-03-30T13:01:41.987920+0000","last_unstale":"2024-03-30T13:04:26.135053+0000","last_undegraded":"2024-03-30T13:04:26.135053+0000","last_fullsized":"2024-03-30T13:04:26.135053+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:03:49.426454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:49.426454+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:10.035429+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22789962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":127,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-30T13:04:27.400270+0000","last_change":"2024-03-30T13:04:27.400348+0000","last_active":"2024-03-30T13:04:27.400270+0000","last_peered":"2024-03-30T13:04:27.400270+0000","last_clean":"2024-03-30T13:04:27.400270+0000","last_became_active":"2024-03-30T13:03:25.178475+0000","last_became_peered":"2024-03-30T13:03:25.178475+0000","last_unstale":"2024-03-30T13:04:27.400270+0000","last_undegraded":"2024-03-30T13:04:27.400270+0000","last_fullsized":"2024-03-30T13:04:27.400270+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:04:27.400094+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:04:27.400094+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:34.847464+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":553,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-30T13:04:27.678820+0000","last_change":"2024-03-30T13:04:16.038106+0000","last_active":"2024-03-30T13:04:27.678820+0000","last_peered":"2024-03-30T13:04:27.678820+0000","last_clean":"2024-03-30T13:04:27.678820+0000","last_became_active":"2024-03-30T13:04:16.037485+0000","last_became_peered":"2024-03-30T13:04:16.037485+0000","last_unstale":"2024-03-30T13:04:27.678820+0000","last_undegraded":"2024-03-30T13:04:27.678820+0000","last_fullsized":"2024-03-30T13:04:27.678820+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:03:57.023686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:57.023686+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:20.958312+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29378240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":612,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:04:26.135209+0000","last_change":"2024-03-30T13:03:30.573752+0000","last_active":"2024-03-30T13:04:26.135209+0000","last_peered":"2024-03-30T13:04:26.135209+0000","last_clean":"2024-03-30T13:04:26.135209+0000","last_became_active":"2024-03-30T13:03:25.980366+0000","last_became_peered":"2024-03-30T13:03:25.980366+0000","last_unstale":"2024-03-30T13:04:26.135209+0000","last_undegraded":"2024-03-30T13:04:26.135209+0000","last_fullsized":"2024-03-30T13:04:26.135209+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:03:30.573489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:30.573489+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:39.840213+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":45680832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":441,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-30T13:04:29.176631+0000","last_change":"2024-03-30T13:04:16.045307+0000","last_active":"2024-03-30T13:04:29.176631+0000","last_peered":"2024-03-30T13:04:29.176631+0000","last_clean":"2024-03-30T13:04:29.176631+0000","last_became_active":"2024-03-30T13:04:16.043916+0000","last_became_peered":"2024-03-30T13:04:16.043916+0000","last_unstale":"2024-03-30T13:04:29.176631+0000","last_undegraded":"2024-03-30T13:04:29.176631+0000","last_fullsized":"2024-03-30T13:04:29.176631+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:03:44.984484+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:03:44.984484+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:50.797137+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":547,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-30T13:04:29.980536+0000","last_change":"2024-03-30T13:04:16.045773+0000","last_active":"2024-03-30T13:04:29.980536+0000","last_peered":"2024-03-30T13:04:29.980536+0000","last_clean":"2024-03-30T13:04:29.980536+0000","last_became_active":"2024-03-30T13:04:16.044565+0000","last_became_peered":"2024-03-30T13:04:16.044565+0000","last_unstale":"2024-03-30T13:04:29.980536+0000","last_undegraded":"2024-03-30T13:04:29.980536+0000","last_fullsized":"2024-03-30T13:04:29.980536+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:03:46.560815+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:03:46.560815+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:49.820257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":543,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-30T13:04:29.980883+0000","last_change":"2024-03-30T13:04:16.050720+0000","last_active":"2024-03-30T13:04:29.980883+0000","last_peered":"2024-03-30T13:04:29.980883+0000","last_clean":"2024-03-30T13:04:29.980883+0000","last_became_active":"2024-03-30T13:04:16.049244+0000","last_became_peered":"2024-03-30T13:04:16.049244+0000","last_unstale":"2024-03-30T13:04:29.980883+0000","last_undegraded":"2024-03-30T13:04:29.980883+0000","last_fullsized":"2024-03-30T13:04:29.980883+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:03:28.655753+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:52.464380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":204,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-30T13:04:29.980157+0000","last_change":"2024-03-30T13:04:21.312886+0000","last_active":"2024-03-30T13:04:29.980157+0000","last_peered":"2024-03-30T13:04:29.980157+0000","last_clean":"2024-03-30T13:04:29.980157+0000","last_became_active":"2024-03-30T13:01:41.982566+0000","last_became_peered":"2024-03-30T13:01:41.982566+0000","last_unstale":"2024-03-30T13:04:29.980157+0000","last_undegraded":"2024-03-30T13:04:29.980157+0000","last_fullsized":"2024-03-30T13:04:29.980157+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_clean_scrub_stamp":"2024-03-30T13:04:21.312610+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:50.841034+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":96,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-30T13:04:27.395097+0000","last_change":"2024-03-30T13:03:32.547779+0000","last_active":"2024-03-30T13:04:27.395097+0000","last_peered":"2024-03-30T13:04:27.395097+0000","last_clean":"2024-03-30T13:04:27.395097+0000","last_became_active":"2024-03-30T13:03:25.178234+0000","last_became_peered":"2024-03-30T13:03:25.178234+0000","last_unstale":"2024-03-30T13:04:27.395097+0000","last_undegraded":"2024-03-30T13:04:27.395097+0000","last_fullsized":"2024-03-30T13:04:27.395097+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:03:32.547475+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:03:32.547475+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:48.688284+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":226,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-30T13:04:29.980231+0000","last_change":"2024-03-30T13:04:16.040294+0000","last_active":"2024-03-30T13:04:29.980231+0000","last_peered":"2024-03-30T13:04:29.980231+0000","last_clean":"2024-03-30T13:04:29.980231+0000","last_became_active":"2024-03-30T13:04:16.039183+0000","last_became_peered":"2024-03-30T13:04:16.039183+0000","last_unstale":"2024-03-30T13:04:29.980231+0000","last_undegraded":"2024-03-30T13:04:29.980231+0000","last_fullsized":"2024-03-30T13:04:29.980231+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:04:11.411056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:04:11.411056+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:21.985751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":519,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-30T13:04:27.678863+0000","last_change":"2024-03-30T13:03:49.092146+0000","last_active":"2024-03-30T13:04:27.678863+0000","last_peered":"2024-03-30T13:04:27.678863+0000","last_clean":"2024-03-30T13:04:27.678863+0000","last_became_active":"2024-03-30T13:01:41.986001+0000","last_became_peered":"2024-03-30T13:01:41.986001+0000","last_unstale":"2024-03-30T13:04:27.678863+0000","last_undegraded":"2024-03-30T13:04:27.678863+0000","last_fullsized":"2024-03-30T13:04:27.678863+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:03:49.091864+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:03:49.091864+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:51.096108+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10453822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":179126564,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":227,"seq":974957576197,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49628,"kb_used_data":17928,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94322212,"statfs":{"total":96636764160,"available":96585945088,"internally_reserved":0,"allocated":18358272,"data_stored":16758206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":156,"seq":670014898213,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54388,"kb_used_data":22696,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94317452,"statfs":{"total":96636764160,"available":96581070848,"internally_reserved":0,"allocated":23240704,"data_stored":21627036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4405,"internal_metadata":32435915},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":135,"seq":579820585007,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46548,"kb_used_data":14400,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94325292,"statfs":{"total":96636764160,"available":96589099008,"internally_reserved":0,"allocated":14745600,"data_stored":13131932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6047,"internal_metadata":32893025},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509551,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40348,"kb_used_data":9168,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331492,"statfs":{"total":96636764160,"available":96595447808,"internally_reserved":0,"allocated":9388032,"data_stored":7774364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285636,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39660,"kb_used_data":7964,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94332180,"statfs":{"total":96636764160,"available":96596152320,"internally_reserved":0,"allocated":8155136,"data_stored":6568572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509551,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47500,"kb_used_data":16320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324340,"statfs":{"total":96636764160,"available":96588124160,"internally_reserved":0,"allocated":16711680,"data_stored":15111614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":206,"seq":884763262991,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45904,"kb_used_data":12864,"kb_used_omap":9,"kb_used_meta":33014,"kb_avail":94325936,"statfs":{"total":96636764160,"available":96589758464,"internally_reserved":0,"allocated":13172736,"data_stored":11572670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9338,"internal_metadata":33807238},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574959,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42328,"kb_used_data":15308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329512,"statfs":{"total":96636764160,"available":96593420288,"internally_reserved":0,"allocated":15675392,"data_stored":14082027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:04:31.523 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:04:31.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:31.635+0000 7f73b8173640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:31.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:31.647+0000 7f73b8173640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:31.815 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:04:31.815 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:04:31.831 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2024-03-30T13:04:31.186006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":179126564,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":366304,"kb_used_data":116648,"kb_used_omap":32,"kb_used_meta":249503,"kb_avail":754608416,"statfs":{"total":773094113280,"available":772719017984,"internally_reserved":0,"allocated":119447552,"data_stored":106626421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33373,"internal_metadata":255491491},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.031922"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":508,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-30T13:04:29.981035+0000","last_change":"2024-03-30T13:04:16.526573+0000","last_active":"2024-03-30T13:04:29.981035+0000","last_peered":"2024-03-30T13:04:29.981035+0000","last_clean":"2024-03-30T13:04:29.981035+0000","last_became_active":"2024-03-30T13:04:16.044630+0000","last_became_peered":"2024-03-30T13:04:16.044630+0000","last_unstale":"2024-03-30T13:04:29.981035+0000","last_undegraded":"2024-03-30T13:04:29.981035+0000","last_fullsized":"2024-03-30T13:04:29.981035+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:04:16.526295+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:30.116419+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":524,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-30T13:04:29.176735+0000","last_change":"2024-03-30T13:03:51.058827+0000","last_active":"2024-03-30T13:04:29.176735+0000","last_peered":"2024-03-30T13:04:29.176735+0000","last_clean":"2024-03-30T13:04:29.176735+0000","last_became_active":"2024-03-30T13:03:25.181879+0000","last_became_peered":"2024-03-30T13:03:25.181879+0000","last_unstale":"2024-03-30T13:04:29.176735+0000","last_undegraded":"2024-03-30T13:04:29.176735+0000","last_fullsized":"2024-03-30T13:04:29.176735+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:03:51.058584+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:03:51.058584+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:02.988983+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10621892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":100,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-30T13:04:27.395405+0000","last_change":"2024-03-30T13:03:39.460832+0000","last_active":"2024-03-30T13:04:27.395405+0000","last_peered":"2024-03-30T13:04:27.395405+0000","last_clean":"2024-03-30T13:04:27.395405+0000","last_became_active":"2024-03-30T13:03:25.175168+0000","last_became_peered":"2024-03-30T13:03:25.175168+0000","last_unstale":"2024-03-30T13:04:27.395405+0000","last_undegraded":"2024-03-30T13:04:27.395405+0000","last_fullsized":"2024-03-30T13:04:27.395405+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_clean_scrub_stamp":"2024-03-30T13:03:39.460563+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:54.437907+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":116,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-30T13:04:27.395070+0000","last_change":"2024-03-30T13:04:16.040508+0000","last_active":"2024-03-30T13:04:27.395070+0000","last_peered":"2024-03-30T13:04:27.395070+0000","last_clean":"2024-03-30T13:04:27.395070+0000","last_became_active":"2024-03-30T13:04:16.039346+0000","last_became_peered":"2024-03-30T13:04:16.039346+0000","last_unstale":"2024-03-30T13:04:27.395070+0000","last_undegraded":"2024-03-30T13:04:27.395070+0000","last_fullsized":"2024-03-30T13:04:27.395070+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:03:29.559074+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:03:29.559074+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:34.625794+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":190,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-30T13:04:29.980187+0000","last_change":"2024-03-30T13:03:58.424647+0000","last_active":"2024-03-30T13:04:29.980187+0000","last_peered":"2024-03-30T13:04:29.980187+0000","last_clean":"2024-03-30T13:04:29.980187+0000","last_became_active":"2024-03-30T13:01:41.984313+0000","last_became_peered":"2024-03-30T13:01:41.984313+0000","last_unstale":"2024-03-30T13:04:29.980187+0000","last_undegraded":"2024-03-30T13:04:29.980187+0000","last_fullsized":"2024-03-30T13:04:29.980187+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:03:58.424408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:58.424408+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:16.577293+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":541,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:04:26.135053+0000","last_change":"2024-03-30T13:03:49.426733+0000","last_active":"2024-03-30T13:04:26.135053+0000","last_peered":"2024-03-30T13:04:26.135053+0000","last_clean":"2024-03-30T13:04:26.135053+0000","last_became_active":"2024-03-30T13:01:41.987920+0000","last_became_peered":"2024-03-30T13:01:41.987920+0000","last_unstale":"2024-03-30T13:04:26.135053+0000","last_undegraded":"2024-03-30T13:04:26.135053+0000","last_fullsized":"2024-03-30T13:04:26.135053+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:03:49.426454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:49.426454+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:10.035429+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22789962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":127,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-30T13:04:27.400270+0000","last_change":"2024-03-30T13:04:27.400348+0000","last_active":"2024-03-30T13:04:27.400270+0000","last_peered":"2024-03-30T13:04:27.400270+0000","last_clean":"2024-03-30T13:04:27.400270+0000","last_became_active":"2024-03-30T13:03:25.178475+0000","last_became_peered":"2024-03-30T13:03:25.178475+0000","last_unstale":"2024-03-30T13:04:27.400270+0000","last_undegraded":"2024-03-30T13:04:27.400270+0000","last_fullsized":"2024-03-30T13:04:27.400270+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:04:27.400094+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:04:27.400094+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:34.847464+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":553,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-30T13:04:27.678820+0000","last_change":"2024-03-30T13:04:16.038106+0000","last_active":"2024-03-30T13:04:27.678820+0000","last_peered":"2024-03-30T13:04:27.678820+0000","last_clean":"2024-03-30T13:04:27.678820+0000","last_became_active":"2024-03-30T13:04:16.037485+0000","last_became_peered":"2024-03-30T13:04:16.037485+0000","last_unstale":"2024-03-30T13:04:27.678820+0000","last_undegraded":"2024-03-30T13:04:27.678820+0000","last_fullsized":"2024-03-30T13:04:27.678820+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:03:57.023686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:57.023686+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:20.958312+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29378240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":612,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:04:26.135209+0000","last_change":"2024-03-30T13:03:30.573752+0000","last_active":"2024-03-30T13:04:26.135209+0000","last_peered":"2024-03-30T13:04:26.135209+0000","last_clean":"2024-03-30T13:04:26.135209+0000","last_became_active":"2024-03-30T13:03:25.980366+0000","last_became_peered":"2024-03-30T13:03:25.980366+0000","last_unstale":"2024-03-30T13:04:26.135209+0000","last_undegraded":"2024-03-30T13:04:26.135209+0000","last_fullsized":"2024-03-30T13:04:26.135209+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:03:30.573489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:30.573489+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:39.840213+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":45680832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":441,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-30T13:04:29.176631+0000","last_change":"2024-03-30T13:04:16.045307+0000","last_active":"2024-03-30T13:04:29.176631+0000","last_peered":"2024-03-30T13:04:29.176631+0000","last_clean":"2024-03-30T13:04:29.176631+0000","last_became_active":"2024-03-30T13:04:16.043916+0000","last_became_peered":"2024-03-30T13:04:16.043916+0000","last_unstale":"2024-03-30T13:04:29.176631+0000","last_undegraded":"2024-03-30T13:04:29.176631+0000","last_fullsized":"2024-03-30T13:04:29.176631+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:03:44.984484+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:03:44.984484+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:50.797137+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":547,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-30T13:04:29.980536+0000","last_change":"2024-03-30T13:04:16.045773+0000","last_active":"2024-03-30T13:04:29.980536+0000","last_peered":"2024-03-30T13:04:29.980536+0000","last_clean":"2024-03-30T13:04:29.980536+0000","last_became_active":"2024-03-30T13:04:16.044565+0000","last_became_peered":"2024-03-30T13:04:16.044565+0000","last_unstale":"2024-03-30T13:04:29.980536+0000","last_undegraded":"2024-03-30T13:04:29.980536+0000","last_fullsized":"2024-03-30T13:04:29.980536+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:03:46.560815+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:03:46.560815+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:49.820257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":543,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-30T13:04:29.980883+0000","last_change":"2024-03-30T13:04:16.050720+0000","last_active":"2024-03-30T13:04:29.980883+0000","last_peered":"2024-03-30T13:04:29.980883+0000","last_clean":"2024-03-30T13:04:29.980883+0000","last_became_active":"2024-03-30T13:04:16.049244+0000","last_became_peered":"2024-03-30T13:04:16.049244+0000","last_unstale":"2024-03-30T13:04:29.980883+0000","last_undegraded":"2024-03-30T13:04:29.980883+0000","last_fullsized":"2024-03-30T13:04:29.980883+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:03:28.655753+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:52.464380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":204,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-30T13:04:29.980157+0000","last_change":"2024-03-30T13:04:21.312886+0000","last_active":"2024-03-30T13:04:29.980157+0000","last_peered":"2024-03-30T13:04:29.980157+0000","last_clean":"2024-03-30T13:04:29.980157+0000","last_became_active":"2024-03-30T13:01:41.982566+0000","last_became_peered":"2024-03-30T13:01:41.982566+0000","last_unstale":"2024-03-30T13:04:29.980157+0000","last_undegraded":"2024-03-30T13:04:29.980157+0000","last_fullsized":"2024-03-30T13:04:29.980157+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_clean_scrub_stamp":"2024-03-30T13:04:21.312610+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:50.841034+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":96,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-30T13:04:27.395097+0000","last_change":"2024-03-30T13:03:32.547779+0000","last_active":"2024-03-30T13:04:27.395097+0000","last_peered":"2024-03-30T13:04:27.395097+0000","last_clean":"2024-03-30T13:04:27.395097+0000","last_became_active":"2024-03-30T13:03:25.178234+0000","last_became_peered":"2024-03-30T13:03:25.178234+0000","last_unstale":"2024-03-30T13:04:27.395097+0000","last_undegraded":"2024-03-30T13:04:27.395097+0000","last_fullsized":"2024-03-30T13:04:27.395097+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:03:32.547475+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:03:32.547475+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:48.688284+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":226,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-30T13:04:29.980231+0000","last_change":"2024-03-30T13:04:16.040294+0000","last_active":"2024-03-30T13:04:29.980231+0000","last_peered":"2024-03-30T13:04:29.980231+0000","last_clean":"2024-03-30T13:04:29.980231+0000","last_became_active":"2024-03-30T13:04:16.039183+0000","last_became_peered":"2024-03-30T13:04:16.039183+0000","last_unstale":"2024-03-30T13:04:29.980231+0000","last_undegraded":"2024-03-30T13:04:29.980231+0000","last_fullsized":"2024-03-30T13:04:29.980231+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:04:11.411056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:04:11.411056+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:21.985751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":519,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-30T13:04:27.678863+0000","last_change":"2024-03-30T13:03:49.092146+0000","last_active":"2024-03-30T13:04:27.678863+0000","last_peered":"2024-03-30T13:04:27.678863+0000","last_clean":"2024-03-30T13:04:27.678863+0000","last_became_active":"2024-03-30T13:01:41.986001+0000","last_became_peered":"2024-03-30T13:01:41.986001+0000","last_unstale":"2024-03-30T13:04:27.678863+0000","last_undegraded":"2024-03-30T13:04:27.678863+0000","last_fullsized":"2024-03-30T13:04:27.678863+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:03:49.091864+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:03:49.091864+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:51.096108+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10453822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":179126564,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":227,"seq":974957576197,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49628,"kb_used_data":17928,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94322212,"statfs":{"total":96636764160,"available":96585945088,"internally_reserved":0,"allocated":18358272,"data_stored":16758206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":156,"seq":670014898213,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54388,"kb_used_data":22696,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94317452,"statfs":{"total":96636764160,"available":96581070848,"internally_reserved":0,"allocated":23240704,"data_stored":21627036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4405,"internal_metadata":32435915},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":135,"seq":579820585007,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46548,"kb_used_data":14400,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94325292,"statfs":{"total":96636764160,"available":96589099008,"internally_reserved":0,"allocated":14745600,"data_stored":13131932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6047,"internal_metadata":32893025},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509551,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40348,"kb_used_data":9168,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331492,"statfs":{"total":96636764160,"available":96595447808,"internally_reserved":0,"allocated":9388032,"data_stored":7774364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285636,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39660,"kb_used_data":7964,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94332180,"statfs":{"total":96636764160,"available":96596152320,"internally_reserved":0,"allocated":8155136,"data_stored":6568572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509551,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47500,"kb_used_data":16320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324340,"statfs":{"total":96636764160,"available":96588124160,"internally_reserved":0,"allocated":16711680,"data_stored":15111614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":206,"seq":884763262991,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45904,"kb_used_data":12864,"kb_used_omap":9,"kb_used_meta":33014,"kb_avail":94325936,"statfs":{"total":96636764160,"available":96589758464,"internally_reserved":0,"allocated":13172736,"data_stored":11572670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9338,"internal_metadata":33807238},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574959,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42328,"kb_used_data":15308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329512,"statfs":{"total":96636764160,"available":96593420288,"internally_reserved":0,"allocated":15675392,"data_stored":14082027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:04:31.833 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:04:31.833 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:04:31.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:31.943+0000 7f3584c25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:31.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:31.955+0000 7f3584c25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:32.136 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:04:32.136 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":243,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:04:31.182936+0000","last_up_change":"2024-03-30T13:04:15.015778+0000","last_in_change":"2024-03-30T13:04:15.015778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":15,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":174,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":77,"up_from":83,"up_thru":227,"down_at":78,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6818","nonce":3104224893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6821","nonce":3104224893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6828","nonce":3104224893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6825","nonce":3104224893}]},"public_addr":"172.21.15.67:6818/3104224893","cluster_addr":"172.21.15.67:6821/3104224893","heartbeat_back_addr":"172.21.15.67:6828/3104224893","heartbeat_front_addr":"172.21.15.67:6825/3104224893","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":198,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6801","nonce":1504348724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6803","nonce":1504348724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6807","nonce":1504348724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6805","nonce":1504348724}]},"public_addr":"172.21.15.67:6801/1504348724","cluster_addr":"172.21.15.67:6803/1504348724","heartbeat_back_addr":"172.21.15.67:6807/1504348724","heartbeat_front_addr":"172.21.15.67:6805/1504348724","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":36,"last_clean_end":125,"up_from":135,"up_thru":227,"down_at":126,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":2818291404},{"type":"v1","addr":"172.21.15.82:6803","nonce":2818291404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":2818291404},{"type":"v1","addr":"172.21.15.82:6807","nonce":2818291404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":2818291404},{"type":"v1","addr":"172.21.15.82:6815","nonce":2818291404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":2818291404},{"type":"v1","addr":"172.21.15.82:6811","nonce":2818291404}]},"public_addr":"172.21.15.82:6803/2818291404","cluster_addr":"172.21.15.82:6807/2818291404","heartbeat_back_addr":"172.21.15.82:6815/2818291404","heartbeat_front_addr":"172.21.15.82:6811/2818291404","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":151,"up_from":156,"up_thru":227,"down_at":152,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":3428934162},{"type":"v1","addr":"172.21.15.82:6817","nonce":3428934162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":3428934162},{"type":"v1","addr":"172.21.15.82:6819","nonce":3428934162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":3428934162},{"type":"v1","addr":"172.21.15.82:6823","nonce":3428934162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":3428934162},{"type":"v1","addr":"172.21.15.82:6821","nonce":3428934162}]},"public_addr":"172.21.15.82:6817/3428934162","cluster_addr":"172.21.15.82:6819/3428934162","heartbeat_back_addr":"172.21.15.82:6823/3428934162","heartbeat_front_addr":"172.21.15.82:6821/3428934162","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":222,"up_from":227,"up_thru":227,"down_at":223,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3068584906},{"type":"v1","addr":"172.21.15.82:6801","nonce":3068584906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3068584906},{"type":"v1","addr":"172.21.15.82:6806","nonce":3068584906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3068584906},{"type":"v1","addr":"172.21.15.82:6814","nonce":3068584906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3068584906},{"type":"v1","addr":"172.21.15.82:6810","nonce":3068584906}]},"public_addr":"172.21.15.82:6801/3068584906","cluster_addr":"172.21.15.82:6806/3068584906","heartbeat_back_addr":"172.21.15.82:6814/3068584906","heartbeat_front_addr":"172.21.15.82:6810/3068584906","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":180,"last_clean_end":197,"up_from":206,"up_thru":227,"down_at":198,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":1241244807},{"type":"v1","addr":"172.21.15.82:6825","nonce":1241244807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":1241244807},{"type":"v1","addr":"172.21.15.82:6827","nonce":1241244807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":1241244807},{"type":"v1","addr":"172.21.15.82:6831","nonce":1241244807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":1241244807},{"type":"v1","addr":"172.21.15.82:6829","nonce":1241244807}]},"public_addr":"172.21.15.82:6825/1241244807","cluster_addr":"172.21.15.82:6827/1241244807","heartbeat_back_addr":"172.21.15.82:6831/1241244807","heartbeat_front_addr":"172.21.15.82:6829/1241244807","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:04:26.479183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:03:12.370486+0000","dead_epoch":78},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:04:15.285792+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:03:13.947730+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:04:26.573314+0000","dead_epoch":126},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:03:08.285965+0000","dead_epoch":152},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:04:14.477530+0000","dead_epoch":223},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:03:58.552136+0000","dead_epoch":198}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:04:32.155 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:04:32.156 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:04:32.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:32.263+0000 7faa03417640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:32.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:32.283+0000 7faa03417640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:32.453 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:04:32.472 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:04:32.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:32.587+0000 7fd33fb46640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:32.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:32.611+0000 7fd33fb46640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:32.786 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:04:32.786 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:04:32.804 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:04:32.804 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:04:32.804 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:04:32.804 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:04:32.804 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 3, 1, 4, 5, 7, 6] 2024-03-30T13:04:32.804 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:04:32.804 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:04:32.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:32.911+0000 7f933fea9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:32.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:32.923+0000 7f933fea9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:33.100 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:04:33.100 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:04:33.118 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":506,"stamp":"2024-03-30T13:04:31.936023+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":179126564,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":366304,"kb_used_data":116648,"kb_used_omap":32,"kb_used_meta":249503,"kb_avail":754608416,"statfs":{"total":773094113280,"available":772719017984,"internally_reserved":0,"allocated":119447552,"data_stored":106626421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":33373,"internal_metadata":255491491},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001079"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":508,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-30T13:04:29.981035+0000","last_change":"2024-03-30T13:04:16.526573+0000","last_active":"2024-03-30T13:04:29.981035+0000","last_peered":"2024-03-30T13:04:29.981035+0000","last_clean":"2024-03-30T13:04:29.981035+0000","last_became_active":"2024-03-30T13:04:16.044630+0000","last_became_peered":"2024-03-30T13:04:16.044630+0000","last_unstale":"2024-03-30T13:04:29.981035+0000","last_undegraded":"2024-03-30T13:04:29.981035+0000","last_fullsized":"2024-03-30T13:04:29.981035+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:04:16.526295+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:30.116419+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":524,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-30T13:04:29.176735+0000","last_change":"2024-03-30T13:03:51.058827+0000","last_active":"2024-03-30T13:04:29.176735+0000","last_peered":"2024-03-30T13:04:29.176735+0000","last_clean":"2024-03-30T13:04:29.176735+0000","last_became_active":"2024-03-30T13:03:25.181879+0000","last_became_peered":"2024-03-30T13:03:25.181879+0000","last_unstale":"2024-03-30T13:04:29.176735+0000","last_undegraded":"2024-03-30T13:04:29.176735+0000","last_fullsized":"2024-03-30T13:04:29.176735+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:03:51.058584+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:03:51.058584+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:02.988983+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10621892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":100,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-30T13:04:27.395405+0000","last_change":"2024-03-30T13:03:39.460832+0000","last_active":"2024-03-30T13:04:27.395405+0000","last_peered":"2024-03-30T13:04:27.395405+0000","last_clean":"2024-03-30T13:04:27.395405+0000","last_became_active":"2024-03-30T13:03:25.175168+0000","last_became_peered":"2024-03-30T13:03:25.175168+0000","last_unstale":"2024-03-30T13:04:27.395405+0000","last_undegraded":"2024-03-30T13:04:27.395405+0000","last_fullsized":"2024-03-30T13:04:27.395405+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_clean_scrub_stamp":"2024-03-30T13:03:39.460563+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:54.437907+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":116,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-30T13:04:27.395070+0000","last_change":"2024-03-30T13:04:16.040508+0000","last_active":"2024-03-30T13:04:27.395070+0000","last_peered":"2024-03-30T13:04:27.395070+0000","last_clean":"2024-03-30T13:04:27.395070+0000","last_became_active":"2024-03-30T13:04:16.039346+0000","last_became_peered":"2024-03-30T13:04:16.039346+0000","last_unstale":"2024-03-30T13:04:27.395070+0000","last_undegraded":"2024-03-30T13:04:27.395070+0000","last_fullsized":"2024-03-30T13:04:27.395070+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:03:29.559074+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:03:29.559074+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:34.625794+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":190,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-30T13:04:29.980187+0000","last_change":"2024-03-30T13:03:58.424647+0000","last_active":"2024-03-30T13:04:29.980187+0000","last_peered":"2024-03-30T13:04:29.980187+0000","last_clean":"2024-03-30T13:04:29.980187+0000","last_became_active":"2024-03-30T13:01:41.984313+0000","last_became_peered":"2024-03-30T13:01:41.984313+0000","last_unstale":"2024-03-30T13:04:29.980187+0000","last_undegraded":"2024-03-30T13:04:29.980187+0000","last_fullsized":"2024-03-30T13:04:29.980187+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:03:58.424408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:58.424408+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:16.577293+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":541,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:04:26.135053+0000","last_change":"2024-03-30T13:03:49.426733+0000","last_active":"2024-03-30T13:04:26.135053+0000","last_peered":"2024-03-30T13:04:26.135053+0000","last_clean":"2024-03-30T13:04:26.135053+0000","last_became_active":"2024-03-30T13:01:41.987920+0000","last_became_peered":"2024-03-30T13:01:41.987920+0000","last_unstale":"2024-03-30T13:04:26.135053+0000","last_undegraded":"2024-03-30T13:04:26.135053+0000","last_fullsized":"2024-03-30T13:04:26.135053+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:03:49.426454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:49.426454+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:10.035429+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22789962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":127,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-30T13:04:27.400270+0000","last_change":"2024-03-30T13:04:27.400348+0000","last_active":"2024-03-30T13:04:27.400270+0000","last_peered":"2024-03-30T13:04:27.400270+0000","last_clean":"2024-03-30T13:04:27.400270+0000","last_became_active":"2024-03-30T13:03:25.178475+0000","last_became_peered":"2024-03-30T13:03:25.178475+0000","last_unstale":"2024-03-30T13:04:27.400270+0000","last_undegraded":"2024-03-30T13:04:27.400270+0000","last_fullsized":"2024-03-30T13:04:27.400270+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:04:27.400094+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:04:27.400094+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:34.847464+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":553,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-30T13:04:27.678820+0000","last_change":"2024-03-30T13:04:16.038106+0000","last_active":"2024-03-30T13:04:27.678820+0000","last_peered":"2024-03-30T13:04:27.678820+0000","last_clean":"2024-03-30T13:04:27.678820+0000","last_became_active":"2024-03-30T13:04:16.037485+0000","last_became_peered":"2024-03-30T13:04:16.037485+0000","last_unstale":"2024-03-30T13:04:27.678820+0000","last_undegraded":"2024-03-30T13:04:27.678820+0000","last_fullsized":"2024-03-30T13:04:27.678820+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:03:57.023686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:57.023686+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:20.958312+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29378240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":612,"reported_epoch":238,"state":"active+clean","last_fresh":"2024-03-30T13:04:26.135209+0000","last_change":"2024-03-30T13:03:30.573752+0000","last_active":"2024-03-30T13:04:26.135209+0000","last_peered":"2024-03-30T13:04:26.135209+0000","last_clean":"2024-03-30T13:04:26.135209+0000","last_became_active":"2024-03-30T13:03:25.980366+0000","last_became_peered":"2024-03-30T13:03:25.980366+0000","last_unstale":"2024-03-30T13:04:26.135209+0000","last_undegraded":"2024-03-30T13:04:26.135209+0000","last_fullsized":"2024-03-30T13:04:26.135209+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:03:30.573489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:30.573489+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:39.840213+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":45680832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":441,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-30T13:04:29.176631+0000","last_change":"2024-03-30T13:04:16.045307+0000","last_active":"2024-03-30T13:04:29.176631+0000","last_peered":"2024-03-30T13:04:29.176631+0000","last_clean":"2024-03-30T13:04:29.176631+0000","last_became_active":"2024-03-30T13:04:16.043916+0000","last_became_peered":"2024-03-30T13:04:16.043916+0000","last_unstale":"2024-03-30T13:04:29.176631+0000","last_undegraded":"2024-03-30T13:04:29.176631+0000","last_fullsized":"2024-03-30T13:04:29.176631+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:03:44.984484+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:03:44.984484+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:50.797137+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":547,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-30T13:04:29.980536+0000","last_change":"2024-03-30T13:04:16.045773+0000","last_active":"2024-03-30T13:04:29.980536+0000","last_peered":"2024-03-30T13:04:29.980536+0000","last_clean":"2024-03-30T13:04:29.980536+0000","last_became_active":"2024-03-30T13:04:16.044565+0000","last_became_peered":"2024-03-30T13:04:16.044565+0000","last_unstale":"2024-03-30T13:04:29.980536+0000","last_undegraded":"2024-03-30T13:04:29.980536+0000","last_fullsized":"2024-03-30T13:04:29.980536+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:03:46.560815+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:03:46.560815+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:49.820257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":543,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-30T13:04:29.980883+0000","last_change":"2024-03-30T13:04:16.050720+0000","last_active":"2024-03-30T13:04:29.980883+0000","last_peered":"2024-03-30T13:04:29.980883+0000","last_clean":"2024-03-30T13:04:29.980883+0000","last_became_active":"2024-03-30T13:04:16.049244+0000","last_became_peered":"2024-03-30T13:04:16.049244+0000","last_unstale":"2024-03-30T13:04:29.980883+0000","last_undegraded":"2024-03-30T13:04:29.980883+0000","last_fullsized":"2024-03-30T13:04:29.980883+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:03:28.655753+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:52.464380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":204,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-30T13:04:29.980157+0000","last_change":"2024-03-30T13:04:21.312886+0000","last_active":"2024-03-30T13:04:29.980157+0000","last_peered":"2024-03-30T13:04:29.980157+0000","last_clean":"2024-03-30T13:04:29.980157+0000","last_became_active":"2024-03-30T13:01:41.982566+0000","last_became_peered":"2024-03-30T13:01:41.982566+0000","last_unstale":"2024-03-30T13:04:29.980157+0000","last_undegraded":"2024-03-30T13:04:29.980157+0000","last_fullsized":"2024-03-30T13:04:29.980157+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_clean_scrub_stamp":"2024-03-30T13:04:21.312610+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:50.841034+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":96,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-30T13:04:27.395097+0000","last_change":"2024-03-30T13:03:32.547779+0000","last_active":"2024-03-30T13:04:27.395097+0000","last_peered":"2024-03-30T13:04:27.395097+0000","last_clean":"2024-03-30T13:04:27.395097+0000","last_became_active":"2024-03-30T13:03:25.178234+0000","last_became_peered":"2024-03-30T13:03:25.178234+0000","last_unstale":"2024-03-30T13:04:27.395097+0000","last_undegraded":"2024-03-30T13:04:27.395097+0000","last_fullsized":"2024-03-30T13:04:27.395097+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:03:32.547475+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:03:32.547475+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:48.688284+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":226,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-30T13:04:29.980231+0000","last_change":"2024-03-30T13:04:16.040294+0000","last_active":"2024-03-30T13:04:29.980231+0000","last_peered":"2024-03-30T13:04:29.980231+0000","last_clean":"2024-03-30T13:04:29.980231+0000","last_became_active":"2024-03-30T13:04:16.039183+0000","last_became_peered":"2024-03-30T13:04:16.039183+0000","last_unstale":"2024-03-30T13:04:29.980231+0000","last_undegraded":"2024-03-30T13:04:29.980231+0000","last_fullsized":"2024-03-30T13:04:29.980231+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:04:11.411056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:04:11.411056+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:21.985751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":519,"reported_epoch":239,"state":"active+clean","last_fresh":"2024-03-30T13:04:27.678863+0000","last_change":"2024-03-30T13:03:49.092146+0000","last_active":"2024-03-30T13:04:27.678863+0000","last_peered":"2024-03-30T13:04:27.678863+0000","last_clean":"2024-03-30T13:04:27.678863+0000","last_became_active":"2024-03-30T13:01:41.986001+0000","last_became_peered":"2024-03-30T13:01:41.986001+0000","last_unstale":"2024-03-30T13:04:27.678863+0000","last_undegraded":"2024-03-30T13:04:27.678863+0000","last_fullsized":"2024-03-30T13:04:27.678863+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:03:49.091864+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:03:49.091864+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:51.096108+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10453822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":179126564,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":227,"seq":974957576197,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49628,"kb_used_data":17928,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94322212,"statfs":{"total":96636764160,"available":96585945088,"internally_reserved":0,"allocated":18358272,"data_stored":16758206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":156,"seq":670014898213,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54388,"kb_used_data":22696,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94317452,"statfs":{"total":96636764160,"available":96581070848,"internally_reserved":0,"allocated":23240704,"data_stored":21627036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4405,"internal_metadata":32435915},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":135,"seq":579820585007,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46548,"kb_used_data":14400,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94325292,"statfs":{"total":96636764160,"available":96589099008,"internally_reserved":0,"allocated":14745600,"data_stored":13131932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6047,"internal_metadata":32893025},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509551,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40348,"kb_used_data":9168,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331492,"statfs":{"total":96636764160,"available":96595447808,"internally_reserved":0,"allocated":9388032,"data_stored":7774364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285636,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39660,"kb_used_data":7964,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94332180,"statfs":{"total":96636764160,"available":96596152320,"internally_reserved":0,"allocated":8155136,"data_stored":6568572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509551,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47500,"kb_used_data":16320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324340,"statfs":{"total":96636764160,"available":96588124160,"internally_reserved":0,"allocated":16711680,"data_stored":15111614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":206,"seq":884763262991,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45904,"kb_used_data":12864,"kb_used_omap":9,"kb_used_meta":33014,"kb_avail":94325936,"statfs":{"total":96636764160,"available":96589758464,"internally_reserved":0,"allocated":13172736,"data_stored":11572670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9338,"internal_metadata":33807238},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574959,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42328,"kb_used_data":15308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329512,"statfs":{"total":96636764160,"available":96593420288,"internally_reserved":0,"allocated":15675392,"data_stored":14082027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:04:33.119 INFO:tasks.thrashosds.thrasher:Choosing PG 1.5 with acting set [4, 2, 6] 2024-03-30T13:04:33.119 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [0, 2, 3, 1, 4, 5, 7, 6] 2024-03-30T13:04:33.119 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-30T13:04:33.120 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:04:33.178 INFO:tasks.ceph.osd.4:Stopped 2024-03-30T13:04:33.178 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 2, 3, 1, 4, 5, 7, 6] 2024-03-30T13:04:33.178 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 4 2024-03-30T13:04:33.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:33.303+0000 7fe250503640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:33.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:33.315+0000 7fe250503640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:33.499 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:04:33.495+0000 7fbb38f08640 -1 mon.a@0(leader).osd e244 definitely_dead 0 2024-03-30T13:04:34.211 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.4. 2024-03-30T13:04:34.239 INFO:tasks.thrashosds.thrasher:dead_osds=[4], live_osds=[0, 2, 3, 1, 5, 7, 6] 2024-03-30T13:04:34.239 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:04:34.239 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:04:34.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:34.355+0000 7f7f8ff22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:34.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:04:34.367+0000 7f7f8ff22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:04:35.388 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:04:35.388 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:04:35.413 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":510,"stamp":"2024-03-30T13:04:35.209682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":163832392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":43,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":320116,"kb_used_data":102608,"kb_used_omap":26,"kb_used_meta":217381,"kb_avail":660282764,"statfs":{"total":676457349120,"available":676129550336,"internally_reserved":0,"allocated":105070592,"data_stored":93660351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27326,"internal_metadata":222598466},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-15294172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.025885"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":514,"reported_epoch":244,"state":"peering","last_fresh":"2024-03-30T13:04:33.211580+0000","last_change":"2024-03-30T13:04:33.211580+0000","last_active":"2024-03-30T13:04:33.211580+0000","last_peered":"2024-03-30T13:04:31.188579+0000","last_clean":"2024-03-30T13:04:31.188579+0000","last_became_active":"2024-03-30T13:04:16.044630+0000","last_became_peered":"2024-03-30T13:04:16.044630+0000","last_unstale":"2024-03-30T13:04:33.211580+0000","last_undegraded":"2024-03-30T13:04:33.211580+0000","last_fullsized":"2024-03-30T13:04:33.211580+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:04:16.526295+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":2,"reported_epoch":245,"state":"peering","last_fresh":"2024-03-30T12:55:47.511068+0000","last_change":"2024-03-30T13:04:34.883398+0000","last_active":"2024-03-30T13:04:34.883398+0000","last_peered":"2024-03-30T12:55:47.511068+0000","last_clean":"2024-03-30T12:55:47.511068+0000","last_became_active":"2024-03-30T12:55:43.930591+0000","last_became_peered":"2024-03-30T12:55:43.930591+0000","last_unstale":"2024-03-30T12:55:47.511068+0000","last_undegraded":"2024-03-30T12:55:47.511068+0000","last_fullsized":"2024-03-30T12:55:47.511068+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:03:51.058584+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:03:51.058584+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":108,"reported_epoch":243,"state":"peering","last_fresh":"2024-03-30T13:04:31.861878+0000","last_change":"2024-03-30T13:04:33.218649+0000","last_active":"2024-03-30T13:04:33.218649+0000","last_peered":"2024-03-30T13:04:31.861878+0000","last_clean":"2024-03-30T13:04:31.861878+0000","last_became_active":"2024-03-30T13:03:25.175168+0000","last_became_peered":"2024-03-30T13:03:25.175168+0000","last_unstale":"2024-03-30T13:04:31.861878+0000","last_undegraded":"2024-03-30T13:04:31.861878+0000","last_fullsized":"2024-03-30T13:04:31.861878+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_clean_scrub_stamp":"2024-03-30T13:03:39.460563+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2147483647,1],"acting":[7,2147483647,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":125,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-03-30T13:04:33.217459+0000","last_change":"2024-03-30T13:04:16.040508+0000","last_active":"2024-03-30T13:04:33.217459+0000","last_peered":"2024-03-30T13:04:33.217459+0000","last_clean":"2024-03-30T13:04:33.217459+0000","last_became_active":"2024-03-30T13:04:16.039346+0000","last_became_peered":"2024-03-30T13:04:16.039346+0000","last_unstale":"2024-03-30T13:04:33.217459+0000","last_undegraded":"2024-03-30T13:04:33.217459+0000","last_fullsized":"2024-03-30T13:04:33.217459+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:03:29.559074+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:03:29.559074+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:34.625794+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":190,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-30T13:04:29.980187+0000","last_change":"2024-03-30T13:03:58.424647+0000","last_active":"2024-03-30T13:04:29.980187+0000","last_peered":"2024-03-30T13:04:29.980187+0000","last_clean":"2024-03-30T13:04:29.980187+0000","last_became_active":"2024-03-30T13:01:41.984313+0000","last_became_peered":"2024-03-30T13:01:41.984313+0000","last_unstale":"2024-03-30T13:04:29.980187+0000","last_undegraded":"2024-03-30T13:04:29.980187+0000","last_fullsized":"2024-03-30T13:04:29.980187+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:03:58.424408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:58.424408+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:16.577293+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":551,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-30T13:04:32.405953+0000","last_change":"2024-03-30T13:03:49.426733+0000","last_active":"2024-03-30T13:04:32.405953+0000","last_peered":"2024-03-30T13:04:32.405953+0000","last_clean":"2024-03-30T13:04:32.405953+0000","last_became_active":"2024-03-30T13:01:41.987920+0000","last_became_peered":"2024-03-30T13:01:41.987920+0000","last_unstale":"2024-03-30T13:04:32.405953+0000","last_undegraded":"2024-03-30T13:04:32.405953+0000","last_fullsized":"2024-03-30T13:04:32.405953+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:03:49.426454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:49.426454+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:10.035429+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22789962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":135,"reported_epoch":243,"state":"peering","last_fresh":"2024-03-30T13:04:31.861924+0000","last_change":"2024-03-30T13:04:33.218007+0000","last_active":"2024-03-30T13:04:33.218007+0000","last_peered":"2024-03-30T13:04:31.861924+0000","last_clean":"2024-03-30T13:04:31.861924+0000","last_became_active":"2024-03-30T13:03:25.178475+0000","last_became_peered":"2024-03-30T13:03:25.178475+0000","last_unstale":"2024-03-30T13:04:31.861924+0000","last_undegraded":"2024-03-30T13:04:31.861924+0000","last_fullsized":"2024-03-30T13:04:31.861924+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:04:27.400094+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:04:27.400094+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2147483647],"acting":[7,0,2147483647],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":561,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-30T13:04:31.677442+0000","last_change":"2024-03-30T13:04:16.038106+0000","last_active":"2024-03-30T13:04:31.677442+0000","last_peered":"2024-03-30T13:04:31.677442+0000","last_clean":"2024-03-30T13:04:31.677442+0000","last_became_active":"2024-03-30T13:04:16.037485+0000","last_became_peered":"2024-03-30T13:04:16.037485+0000","last_unstale":"2024-03-30T13:04:31.677442+0000","last_undegraded":"2024-03-30T13:04:31.677442+0000","last_fullsized":"2024-03-30T13:04:31.677442+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:03:57.023686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:57.023686+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:20.958312+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29378240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":622,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-30T13:04:32.405929+0000","last_change":"2024-03-30T13:03:30.573752+0000","last_active":"2024-03-30T13:04:32.405929+0000","last_peered":"2024-03-30T13:04:32.405929+0000","last_clean":"2024-03-30T13:04:32.405929+0000","last_became_active":"2024-03-30T13:03:25.980366+0000","last_became_peered":"2024-03-30T13:03:25.980366+0000","last_unstale":"2024-03-30T13:04:32.405929+0000","last_undegraded":"2024-03-30T13:04:32.405929+0000","last_fullsized":"2024-03-30T13:04:32.405929+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:03:30.573489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:30.573489+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:39.840213+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":45680832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":2,"reported_epoch":245,"state":"peering","last_fresh":"2024-03-30T13:04:33.208233+0000","last_change":"2024-03-30T12:56:43.118093+0000","last_active":"2024-03-30T12:56:43.118093+0000","last_peered":"2024-03-30T12:55:47.249216+0000","last_clean":"2024-03-30T12:55:47.249216+0000","last_became_active":"2024-03-30T12:55:43.936547+0000","last_became_peered":"2024-03-30T12:55:43.936547+0000","last_unstale":"2024-03-30T13:04:33.208233+0000","last_undegraded":"2024-03-30T13:04:33.208233+0000","last_fullsized":"2024-03-30T13:04:33.208233+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:03:44.984484+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:03:44.984484+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":553,"reported_epoch":244,"state":"peering","last_fresh":"2024-03-30T13:04:33.211824+0000","last_change":"2024-03-30T13:04:33.211824+0000","last_active":"2024-03-30T13:04:33.211824+0000","last_peered":"2024-03-30T13:04:31.188404+0000","last_clean":"2024-03-30T13:04:31.188404+0000","last_became_active":"2024-03-30T13:04:16.044565+0000","last_became_peered":"2024-03-30T13:04:16.044565+0000","last_unstale":"2024-03-30T13:04:33.211824+0000","last_undegraded":"2024-03-30T13:04:33.211824+0000","last_fullsized":"2024-03-30T13:04:33.211824+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:03:46.560815+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:03:46.560815+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":551,"reported_epoch":245,"state":"active+clean","last_fresh":"2024-03-30T13:04:34.214235+0000","last_change":"2024-03-30T13:04:16.050720+0000","last_active":"2024-03-30T13:04:34.214235+0000","last_peered":"2024-03-30T13:04:34.214235+0000","last_clean":"2024-03-30T13:04:34.214235+0000","last_became_active":"2024-03-30T13:04:16.049244+0000","last_became_peered":"2024-03-30T13:04:16.049244+0000","last_unstale":"2024-03-30T13:04:34.214235+0000","last_undegraded":"2024-03-30T13:04:34.214235+0000","last_fullsized":"2024-03-30T13:04:34.214235+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:03:28.655753+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:52.464380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":204,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-30T13:04:29.980157+0000","last_change":"2024-03-30T13:04:21.312886+0000","last_active":"2024-03-30T13:04:29.980157+0000","last_peered":"2024-03-30T13:04:29.980157+0000","last_clean":"2024-03-30T13:04:29.980157+0000","last_became_active":"2024-03-30T13:01:41.982566+0000","last_became_peered":"2024-03-30T13:01:41.982566+0000","last_unstale":"2024-03-30T13:04:29.980157+0000","last_undegraded":"2024-03-30T13:04:29.980157+0000","last_fullsized":"2024-03-30T13:04:29.980157+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_clean_scrub_stamp":"2024-03-30T13:04:21.312610+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:50.841034+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":105,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-03-30T13:04:33.217556+0000","last_change":"2024-03-30T13:03:32.547779+0000","last_active":"2024-03-30T13:04:33.217556+0000","last_peered":"2024-03-30T13:04:33.217556+0000","last_clean":"2024-03-30T13:04:33.217556+0000","last_became_active":"2024-03-30T13:03:25.178234+0000","last_became_peered":"2024-03-30T13:03:25.178234+0000","last_unstale":"2024-03-30T13:04:33.217556+0000","last_undegraded":"2024-03-30T13:04:33.217556+0000","last_fullsized":"2024-03-30T13:04:33.217556+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:03:32.547475+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:03:32.547475+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:48.688284+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":226,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-30T13:04:29.980231+0000","last_change":"2024-03-30T13:04:16.040294+0000","last_active":"2024-03-30T13:04:29.980231+0000","last_peered":"2024-03-30T13:04:29.980231+0000","last_clean":"2024-03-30T13:04:29.980231+0000","last_became_active":"2024-03-30T13:04:16.039183+0000","last_became_peered":"2024-03-30T13:04:16.039183+0000","last_unstale":"2024-03-30T13:04:29.980231+0000","last_undegraded":"2024-03-30T13:04:29.980231+0000","last_fullsized":"2024-03-30T13:04:29.980231+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:04:11.411056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:04:11.411056+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:21.985751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":527,"reported_epoch":243,"state":"active+clean","last_fresh":"2024-03-30T13:04:31.677587+0000","last_change":"2024-03-30T13:03:49.092146+0000","last_active":"2024-03-30T13:04:31.677587+0000","last_peered":"2024-03-30T13:04:31.677587+0000","last_clean":"2024-03-30T13:04:31.677587+0000","last_became_active":"2024-03-30T13:01:41.986001+0000","last_became_peered":"2024-03-30T13:01:41.986001+0000","last_unstale":"2024-03-30T13:04:31.677587+0000","last_undegraded":"2024-03-30T13:04:31.677587+0000","last_fullsized":"2024-03-30T13:04:31.677587+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:03:49.091864+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:03:49.091864+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:51.096108+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10453822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":163832392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":227,"seq":974957576198,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49696,"kb_used_data":18000,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94322144,"statfs":{"total":96636764160,"available":96585875456,"internally_reserved":0,"allocated":18432000,"data_stored":16791428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":156,"seq":670014898213,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54388,"kb_used_data":22696,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94317452,"statfs":{"total":96636764160,"available":96581070848,"internally_reserved":0,"allocated":23240704,"data_stored":21627036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4405,"internal_metadata":32435915},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509552,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40400,"kb_used_data":9216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331440,"statfs":{"total":96636764160,"available":96595394560,"internally_reserved":0,"allocated":9437184,"data_stored":7796612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285637,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39728,"kb_used_data":8036,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94332112,"statfs":{"total":96636764160,"available":96596082688,"internally_reserved":0,"allocated":8228864,"data_stored":6601594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509552,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47564,"kb_used_data":16380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324276,"statfs":{"total":96636764160,"available":96588058624,"internally_reserved":0,"allocated":16773120,"data_stored":15139275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":206,"seq":884763262992,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45964,"kb_used_data":12924,"kb_used_omap":9,"kb_used_meta":33014,"kb_avail":94325876,"statfs":{"total":96636764160,"available":96589697024,"internally_reserved":0,"allocated":13234176,"data_stored":11600331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9338,"internal_metadata":33807238},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574960,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42376,"kb_used_data":15356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329464,"statfs":{"total":96636764160,"available":96593371136,"internally_reserved":0,"allocated":15724544,"data_stored":14104075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:04:35.414 INFO:tasks.ceph.ceph_manager.ceph:PG 1.f is not active or peered 2024-03-30T13:04:35.414 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.f', 'version': "22'6", 'reported_seq': 514, 'reported_epoch': 244, 'state': 'peering', 'last_fresh': '2024-03-30T13:04:33.211580+0000', 'last_change': '2024-03-30T13:04:33.211580+0000', 'last_active': '2024-03-30T13:04:33.211580+0000', 'last_peered': '2024-03-30T13:04:31.188579+0000', 'last_clean': '2024-03-30T13:04:31.188579+0000', 'last_became_active': '2024-03-30T13:04:16.044630+0000', 'last_became_peered': '2024-03-30T13:04:16.044630+0000', 'last_unstale': '2024-03-30T13:04:33.211580+0000', 'last_undegraded': '2024-03-30T13:04:33.211580+0000', 'last_fullsized': '2024-03-30T13:04:33.211580+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 228, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "22'6", 'last_scrub_stamp': '2024-03-30T13:04:16.526295+0000', 'last_deep_scrub': "22'6", 'last_deep_scrub_stamp': '2024-03-30T13:04:16.526295+0000', 'last_clean_scrub_stamp': '2024-03-30T13:04:16.526295+0000', 'objects_scrubbed': 2, 'log_size': 6, 'log_dups_size': 0, 'ondisk_log_size': 6, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 7, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4588245, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 3, 'num_read_kb': 1, 'num_write': 8, 'num_write_kb': 1781, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4588245, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5, 1], 'acting': [6, 5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1, 5], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-30T13:04:35.414 INFO:tasks.ceph.ceph_manager.ceph:PG 1.e is not active or peered 2024-03-30T13:04:35.414 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.e', 'version': "22'3", 'reported_seq': 2, 'reported_epoch': 245, 'state': 'peering', 'last_fresh': '2024-03-30T12:55:47.511068+0000', 'last_change': '2024-03-30T13:04:34.883398+0000', 'last_active': '2024-03-30T13:04:34.883398+0000', 'last_peered': '2024-03-30T12:55:47.511068+0000', 'last_clean': '2024-03-30T12:55:47.511068+0000', 'last_became_active': '2024-03-30T12:55:43.930591+0000', 'last_became_peered': '2024-03-30T12:55:43.930591+0000', 'last_unstale': '2024-03-30T12:55:47.511068+0000', 'last_undegraded': '2024-03-30T12:55:47.511068+0000', 'last_fullsized': '2024-03-30T12:55:47.511068+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 207, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "22'3", 'last_scrub_stamp': '2024-03-30T13:03:51.058584+0000', 'last_deep_scrub': "22'3", 'last_deep_scrub_stamp': '2024-03-30T12:58:30.264658+0000', 'last_clean_scrub_stamp': '2024-03-30T13:03:51.058584+0000', 'objects_scrubbed': 0, 'log_size': 3, 'log_dups_size': 0, 'ondisk_log_size': 3, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2655473, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 3, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 0, 'num_write': 4, 'num_write_kb': 1244, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7, 3], 'acting': [2, 7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-30T13:04:35.415 INFO:tasks.ceph.ceph_manager.ceph:PG 1.d is not active or peered 2024-03-30T13:04:35.415 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.d', 'version': "22'9", 'reported_seq': 108, 'reported_epoch': 243, 'state': 'peering', 'last_fresh': '2024-03-30T13:04:31.861878+0000', 'last_change': '2024-03-30T13:04:33.218649+0000', 'last_active': '2024-03-30T13:04:33.218649+0000', 'last_peered': '2024-03-30T13:04:31.861878+0000', 'last_clean': '2024-03-30T13:04:31.861878+0000', 'last_became_active': '2024-03-30T13:03:25.175168+0000', 'last_became_peered': '2024-03-30T13:03:25.175168+0000', 'last_unstale': '2024-03-30T13:04:31.861878+0000', 'last_undegraded': '2024-03-30T13:04:31.861878+0000', 'last_fullsized': '2024-03-30T13:04:31.861878+0000', 'mapping_epoch': 244, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 207, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "22'9", 'last_scrub_stamp': '2024-03-30T13:03:39.460563+0000', 'last_deep_scrub': "22'9", 'last_deep_scrub_stamp': '2024-03-30T13:03:39.460563+0000', 'last_clean_scrub_stamp': '2024-03-30T13:03:39.460563+0000', 'objects_scrubbed': 3, 'log_size': 9, 'log_dups_size': 0, 'ondisk_log_size': 9, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 15, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 7815156, 'num_objects': 3, 'num_object_clones': 0, 'num_object_copies': 9, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 3, 'num_whiteouts': 0, 'num_read': 5, 'num_read_kb': 2, 'num_write': 12, 'num_write_kb': 3459, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2147483647, 1], 'acting': [7, 2147483647, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-30T13:04:35.415 INFO:tasks.ceph.ceph_manager.ceph:PG 1.2 is not active or peered 2024-03-30T13:04:35.415 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.2', 'version': "22'18", 'reported_seq': 135, 'reported_epoch': 243, 'state': 'peering', 'last_fresh': '2024-03-30T13:04:31.861924+0000', 'last_change': '2024-03-30T13:04:33.218007+0000', 'last_active': '2024-03-30T13:04:33.218007+0000', 'last_peered': '2024-03-30T13:04:31.861924+0000', 'last_clean': '2024-03-30T13:04:31.861924+0000', 'last_became_active': '2024-03-30T13:03:25.178475+0000', 'last_became_peered': '2024-03-30T13:03:25.178475+0000', 'last_unstale': '2024-03-30T13:04:31.861924+0000', 'last_undegraded': '2024-03-30T13:04:31.861924+0000', 'last_fullsized': '2024-03-30T13:04:31.861924+0000', 'mapping_epoch': 244, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 207, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "22'18", 'last_scrub_stamp': '2024-03-30T13:04:27.400094+0000', 'last_deep_scrub': "22'18", 'last_deep_scrub_stamp': '2024-03-30T13:00:34.155844+0000', 'last_clean_scrub_stamp': '2024-03-30T13:04:27.400094+0000', 'objects_scrubbed': 6, 'log_size': 18, 'log_dups_size': 0, 'ondisk_log_size': 18, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 11, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 14638699, 'num_objects': 6, 'num_object_clones': 0, 'num_object_copies': 18, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6, 'num_whiteouts': 0, 'num_read': 11, 'num_read_kb': 5, 'num_write': 24, 'num_write_kb': 6991, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6, 'num_bytes_recovered': 14638699, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0, 2147483647], 'acting': [7, 0, 2147483647], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-30T13:04:35.415 INFO:tasks.ceph.ceph_manager.ceph:PG 1.5 is not active or peered 2024-03-30T13:04:35.415 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.5', 'version': "22'3", 'reported_seq': 2, 'reported_epoch': 245, 'state': 'peering', 'last_fresh': '2024-03-30T13:04:33.208233+0000', 'last_change': '2024-03-30T12:56:43.118093+0000', 'last_active': '2024-03-30T12:56:43.118093+0000', 'last_peered': '2024-03-30T12:55:47.249216+0000', 'last_clean': '2024-03-30T12:55:47.249216+0000', 'last_became_active': '2024-03-30T12:55:43.936547+0000', 'last_became_peered': '2024-03-30T12:55:43.936547+0000', 'last_unstale': '2024-03-30T13:04:33.208233+0000', 'last_undegraded': '2024-03-30T13:04:33.208233+0000', 'last_fullsized': '2024-03-30T13:04:33.208233+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 228, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "22'3", 'last_scrub_stamp': '2024-03-30T13:03:44.984484+0000', 'last_deep_scrub': "22'3", 'last_deep_scrub_stamp': '2024-03-30T12:59:42.816452+0000', 'last_clean_scrub_stamp': '2024-03-30T13:03:44.984484+0000', 'objects_scrubbed': 0, 'log_size': 3, 'log_dups_size': 0, 'ondisk_log_size': 3, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2336140, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 3, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 0, 'num_write': 4, 'num_write_kb': 835, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2, 6], 'acting': [0, 2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-30T13:04:35.415 INFO:tasks.ceph.ceph_manager.ceph:PG 1.6 is not active or peered 2024-03-30T13:04:35.415 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.6', 'version': "22'15", 'reported_seq': 553, 'reported_epoch': 244, 'state': 'peering', 'last_fresh': '2024-03-30T13:04:33.211824+0000', 'last_change': '2024-03-30T13:04:33.211824+0000', 'last_active': '2024-03-30T13:04:33.211824+0000', 'last_peered': '2024-03-30T13:04:31.188404+0000', 'last_clean': '2024-03-30T13:04:31.188404+0000', 'last_became_active': '2024-03-30T13:04:16.044565+0000', 'last_became_peered': '2024-03-30T13:04:16.044565+0000', 'last_unstale': '2024-03-30T13:04:33.211824+0000', 'last_undegraded': '2024-03-30T13:04:33.211824+0000', 'last_fullsized': '2024-03-30T13:04:33.211824+0000', 'mapping_epoch': 245, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 228, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "22'15", 'last_scrub_stamp': '2024-03-30T13:03:46.560815+0000', 'last_deep_scrub': "22'15", 'last_deep_scrub_stamp': '2024-03-30T13:02:18.933526+0000', 'last_clean_scrub_stamp': '2024-03-30T13:03:46.560815+0000', 'objects_scrubbed': 0, 'log_size': 15, 'log_dups_size': 0, 'ondisk_log_size': 15, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12739541, 'num_objects': 5, 'num_object_clones': 0, 'num_object_copies': 15, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5, 'num_whiteouts': 0, 'num_read': 9, 'num_read_kb': 4, 'num_write': 20, 'num_write_kb': 6289, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0, 3], 'acting': [6, 0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0, 3], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-30T13:04:35.415 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:05:00.417 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:05:00.417 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:05:00.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:00.543+0000 7f29dd798640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:00.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:00.559+0000 7f29dd798640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:00.736 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:05:00.737 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:05:00.754 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":525,"stamp":"2024-03-30T13:04:59.944150+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":208605646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":332676,"kb_used_data":115204,"kb_used_omap":26,"kb_used_meta":217381,"kb_avail":660270204,"statfs":{"total":676457349120,"available":676116688896,"internally_reserved":0,"allocated":117968896,"data_stored":106342827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27326,"internal_metadata":222598466},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003440"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":533,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-30T13:04:40.461436+0000","last_change":"2024-03-30T13:04:35.290046+0000","last_active":"2024-03-30T13:04:40.461436+0000","last_peered":"2024-03-30T13:04:40.461436+0000","last_clean":"2024-03-30T13:04:40.461436+0000","last_became_active":"2024-03-30T13:04:35.229021+0000","last_became_peered":"2024-03-30T13:04:35.229021+0000","last_unstale":"2024-03-30T13:04:40.461436+0000","last_undegraded":"2024-03-30T13:04:40.461436+0000","last_fullsized":"2024-03-30T13:04:40.461436+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:04:16.526295+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:29.532040+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":19,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-30T13:04:40.462782+0000","last_change":"2024-03-30T13:04:35.273862+0000","last_active":"2024-03-30T13:04:40.462782+0000","last_peered":"2024-03-30T13:04:40.462782+0000","last_clean":"2024-03-30T13:04:40.462782+0000","last_became_active":"2024-03-30T13:04:35.222638+0000","last_became_peered":"2024-03-30T13:04:35.222638+0000","last_unstale":"2024-03-30T13:04:40.462782+0000","last_undegraded":"2024-03-30T13:04:40.462782+0000","last_fullsized":"2024-03-30T13:04:40.462782+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:03:51.058584+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:03:51.058584+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:18.077836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":138,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-30T13:04:55.327805+0000","last_change":"2024-03-30T13:04:55.327971+0000","last_active":"2024-03-30T13:04:55.327805+0000","last_peered":"2024-03-30T13:04:55.327805+0000","last_clean":"2024-03-30T13:04:55.327805+0000","last_became_active":"2024-03-30T13:04:35.231335+0000","last_became_peered":"2024-03-30T13:04:35.231335+0000","last_unstale":"2024-03-30T13:04:55.327805+0000","last_undegraded":"2024-03-30T13:04:55.327805+0000","last_fullsized":"2024-03-30T13:04:55.327805+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:04:55.327587+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_clean_scrub_stamp":"2024-03-30T13:04:55.327587+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:16.098367+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":141,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-30T13:04:40.327864+0000","last_change":"2024-03-30T13:04:36.430290+0000","last_active":"2024-03-30T13:04:40.327864+0000","last_peered":"2024-03-30T13:04:40.327864+0000","last_clean":"2024-03-30T13:04:40.327864+0000","last_became_active":"2024-03-30T13:04:16.039346+0000","last_became_peered":"2024-03-30T13:04:16.039346+0000","last_unstale":"2024-03-30T13:04:40.327864+0000","last_undegraded":"2024-03-30T13:04:40.327864+0000","last_fullsized":"2024-03-30T13:04:40.327864+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:36.430045+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:04:36.430045+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:04.050679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":204,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-30T13:04:40.323362+0000","last_change":"2024-03-30T13:03:58.424647+0000","last_active":"2024-03-30T13:04:40.323362+0000","last_peered":"2024-03-30T13:04:40.323362+0000","last_clean":"2024-03-30T13:04:40.323362+0000","last_became_active":"2024-03-30T13:01:41.984313+0000","last_became_peered":"2024-03-30T13:01:41.984313+0000","last_unstale":"2024-03-30T13:04:40.323362+0000","last_undegraded":"2024-03-30T13:04:40.323362+0000","last_fullsized":"2024-03-30T13:04:40.323362+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:03:58.424408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:58.424408+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:16.577293+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":562,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-30T13:04:40.267674+0000","last_change":"2024-03-30T13:03:49.426733+0000","last_active":"2024-03-30T13:04:40.267674+0000","last_peered":"2024-03-30T13:04:40.267674+0000","last_clean":"2024-03-30T13:04:40.267674+0000","last_became_active":"2024-03-30T13:01:41.987920+0000","last_became_peered":"2024-03-30T13:01:41.987920+0000","last_unstale":"2024-03-30T13:04:40.267674+0000","last_undegraded":"2024-03-30T13:04:40.267674+0000","last_fullsized":"2024-03-30T13:04:40.267674+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:03:49.426454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:49.426454+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:10.035429+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22789962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":163,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-30T13:04:40.327899+0000","last_change":"2024-03-30T13:04:36.206989+0000","last_active":"2024-03-30T13:04:40.327899+0000","last_peered":"2024-03-30T13:04:40.327899+0000","last_clean":"2024-03-30T13:04:40.327899+0000","last_became_active":"2024-03-30T13:04:35.230791+0000","last_became_peered":"2024-03-30T13:04:35.230791+0000","last_unstale":"2024-03-30T13:04:40.327899+0000","last_undegraded":"2024-03-30T13:04:40.327899+0000","last_fullsized":"2024-03-30T13:04:40.327899+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:04:27.400094+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:04:27.400094+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:30.147768+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":571,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-30T13:04:40.267921+0000","last_change":"2024-03-30T13:04:16.038106+0000","last_active":"2024-03-30T13:04:40.267921+0000","last_peered":"2024-03-30T13:04:40.267921+0000","last_clean":"2024-03-30T13:04:40.267921+0000","last_became_active":"2024-03-30T13:04:16.037485+0000","last_became_peered":"2024-03-30T13:04:16.037485+0000","last_unstale":"2024-03-30T13:04:40.267921+0000","last_undegraded":"2024-03-30T13:04:40.267921+0000","last_fullsized":"2024-03-30T13:04:40.267921+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:03:57.023686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:57.023686+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:20.958312+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29378240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":640,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-30T13:04:40.339475+0000","last_change":"2024-03-30T13:04:40.339554+0000","last_active":"2024-03-30T13:04:40.339475+0000","last_peered":"2024-03-30T13:04:40.339475+0000","last_clean":"2024-03-30T13:04:40.339475+0000","last_became_active":"2024-03-30T13:03:25.980366+0000","last_became_peered":"2024-03-30T13:03:25.980366+0000","last_unstale":"2024-03-30T13:04:40.339475+0000","last_undegraded":"2024-03-30T13:04:40.339475+0000","last_fullsized":"2024-03-30T13:04:40.339475+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:40.339270+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:40.339270+0000","last_clean_scrub_stamp":"2024-03-30T13:04:40.339270+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:05.690409+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":45680832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":19,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-30T13:04:40.462038+0000","last_change":"2024-03-30T13:04:35.257993+0000","last_active":"2024-03-30T13:04:40.462038+0000","last_peered":"2024-03-30T13:04:40.462038+0000","last_clean":"2024-03-30T13:04:40.462038+0000","last_became_active":"2024-03-30T13:04:35.226187+0000","last_became_peered":"2024-03-30T13:04:35.226187+0000","last_unstale":"2024-03-30T13:04:40.462038+0000","last_undegraded":"2024-03-30T13:04:40.462038+0000","last_fullsized":"2024-03-30T13:04:40.462038+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:03:44.984484+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:03:44.984484+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:14.330843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":578,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-30T13:04:40.461466+0000","last_change":"2024-03-30T13:04:35.346220+0000","last_active":"2024-03-30T13:04:40.461466+0000","last_peered":"2024-03-30T13:04:40.461466+0000","last_clean":"2024-03-30T13:04:40.461466+0000","last_became_active":"2024-03-30T13:04:35.229489+0000","last_became_peered":"2024-03-30T13:04:35.229489+0000","last_unstale":"2024-03-30T13:04:40.461466+0000","last_undegraded":"2024-03-30T13:04:40.461466+0000","last_fullsized":"2024-03-30T13:04:40.461466+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:03:46.560815+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:03:46.560815+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:04:59.970242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":565,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-30T13:04:53.372897+0000","last_change":"2024-03-30T13:04:53.373086+0000","last_active":"2024-03-30T13:04:53.372897+0000","last_peered":"2024-03-30T13:04:53.372897+0000","last_clean":"2024-03-30T13:04:53.372897+0000","last_became_active":"2024-03-30T13:04:16.049244+0000","last_became_peered":"2024-03-30T13:04:16.049244+0000","last_unstale":"2024-03-30T13:04:53.372897+0000","last_undegraded":"2024-03-30T13:04:53.372897+0000","last_fullsized":"2024-03-30T13:04:53.372897+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:04:53.372665+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:04:53.372665+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:05.305383+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":218,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-30T13:04:40.323353+0000","last_change":"2024-03-30T13:04:21.312886+0000","last_active":"2024-03-30T13:04:40.323353+0000","last_peered":"2024-03-30T13:04:40.323353+0000","last_clean":"2024-03-30T13:04:40.323353+0000","last_became_active":"2024-03-30T13:01:41.982566+0000","last_became_peered":"2024-03-30T13:01:41.982566+0000","last_unstale":"2024-03-30T13:04:40.323353+0000","last_undegraded":"2024-03-30T13:04:40.323353+0000","last_fullsized":"2024-03-30T13:04:40.323353+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_clean_scrub_stamp":"2024-03-30T13:04:21.312610+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:50.841034+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":121,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-30T13:04:51.422093+0000","last_change":"2024-03-30T13:04:51.422166+0000","last_active":"2024-03-30T13:04:51.422093+0000","last_peered":"2024-03-30T13:04:51.422093+0000","last_clean":"2024-03-30T13:04:51.422093+0000","last_became_active":"2024-03-30T13:03:25.178234+0000","last_became_peered":"2024-03-30T13:03:25.178234+0000","last_unstale":"2024-03-30T13:04:51.422093+0000","last_undegraded":"2024-03-30T13:04:51.422093+0000","last_fullsized":"2024-03-30T13:04:51.422093+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:04:51.421910+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:04:51.421910+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:54.940198+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":240,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-30T13:04:40.323392+0000","last_change":"2024-03-30T13:04:16.040294+0000","last_active":"2024-03-30T13:04:40.323392+0000","last_peered":"2024-03-30T13:04:40.323392+0000","last_clean":"2024-03-30T13:04:40.323392+0000","last_became_active":"2024-03-30T13:04:16.039183+0000","last_became_peered":"2024-03-30T13:04:16.039183+0000","last_unstale":"2024-03-30T13:04:40.323392+0000","last_undegraded":"2024-03-30T13:04:40.323392+0000","last_fullsized":"2024-03-30T13:04:40.323392+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:04:11.411056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:04:11.411056+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:21.985751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":545,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-30T13:04:55.118996+0000","last_change":"2024-03-30T13:04:55.119064+0000","last_active":"2024-03-30T13:04:55.118996+0000","last_peered":"2024-03-30T13:04:55.118996+0000","last_clean":"2024-03-30T13:04:55.118996+0000","last_became_active":"2024-03-30T13:01:41.986001+0000","last_became_peered":"2024-03-30T13:01:41.986001+0000","last_unstale":"2024-03-30T13:04:55.118996+0000","last_undegraded":"2024-03-30T13:04:55.118996+0000","last_fullsized":"2024-03-30T13:04:55.118996+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:55.118814+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:04:55.118814+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:17.644874+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10453822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":208605646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":108814336,"data_stored":108814336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":227,"seq":974957576202,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49724,"kb_used_data":18036,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94322116,"statfs":{"total":96636764160,"available":96585846784,"internally_reserved":0,"allocated":18468864,"data_stored":16807997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":156,"seq":670014898218,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54468,"kb_used_data":22780,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94317372,"statfs":{"total":96636764160,"available":96580988928,"internally_reserved":0,"allocated":23326720,"data_stored":21665853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4405,"internal_metadata":32435915},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509557,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47184,"kb_used_data":16008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324656,"statfs":{"total":96636764160,"available":96588447744,"internally_reserved":0,"allocated":16392192,"data_stored":14731325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285642,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40876,"kb_used_data":9188,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94330964,"statfs":{"total":96636764160,"available":96594907136,"internally_reserved":0,"allocated":9408512,"data_stored":7747645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509557,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48116,"kb_used_data":16932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323724,"statfs":{"total":96636764160,"available":96587493376,"internally_reserved":0,"allocated":17338368,"data_stored":15677501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":206,"seq":884763262998,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46004,"kb_used_data":12972,"kb_used_omap":9,"kb_used_meta":33014,"kb_avail":94325836,"statfs":{"total":96636764160,"available":96589656064,"internally_reserved":0,"allocated":13283328,"data_stored":11622461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9338,"internal_metadata":33807238},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574965,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46304,"kb_used_data":19288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94325536,"statfs":{"total":96636764160,"available":96589348864,"internally_reserved":0,"allocated":19750912,"data_stored":18090045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14340096,"data_stored":14340096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6410240,"data_stored":6410240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13393920,"data_stored":13393920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16752640,"data_stored":16752640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:05:00.755 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:05:00.755 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:05:00.755 INFO:tasks.thrashosds.thrasher:Reviving osd 4 2024-03-30T13:05:00.755 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-30T13:05:00.755 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-30T13:05:00.758 INFO:tasks.ceph.osd.4:Started 2024-03-30T13:05:00.758 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-30T13:05:00.825 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:05:00.821+0000 7f84472007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:00.833 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:05:00.829+0000 7f84472007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:00.833 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:05:00.829+0000 7f84472007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:00.885 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T13:05:00.885 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T13:05:00.885 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T13:05:00.886 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T13:05:00.886 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T13:05:00.886 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T13:05:00.886 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-03-30T13:05:00.886 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-03-30T13:05:00.886 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-03-30T13:05:00.886 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-03-30T13:05:00.886 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-03-30T13:05:00.886 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T13:05:00.888 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:05:00.888 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-03-30T13:05:01.383 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:05:01.381+0000 7f84472007c0 -1 Falling back to public interface 2024-03-30T13:05:05.130 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:05:05.129+0000 7f84472007c0 -1 osd.4 243 log_to_monitors true 2024-03-30T13:05:05.430 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:05:05.429+0000 7f843afcd640 -1 osd.4 243 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:05:05.889 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-30T13:05:06.019 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T13:05:06.019 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T13:05:06.019 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T13:05:06.019 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T13:05:06.033 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:05:06.033 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-03-30T13:05:06.033 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 4 2024-03-30T13:05:06.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:06.159+0000 7f897a1ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:06.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:06.175+0000 7f897a1ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:06.359 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:05:06.355+0000 7fbb38f08640 -1 mon.a@0(leader).osd e249 definitely_dead 0 2024-03-30T13:05:06.413 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.4. 2024-03-30T13:05:06.436 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-03-30T13:05:06.437 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:05:06.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:06.547+0000 7f616e51a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:06.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:06.563+0000 7f616e51a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:07.432 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:05:07.458 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:05:07.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:07.575+0000 7fe161850640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:07.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:07.591+0000 7fe161850640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:08.433 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:05:08.457 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:05:08.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:08.575+0000 7fb79264f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:08.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:08.591+0000 7fb79264f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:09.447 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:05:09.474 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:05:09.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:09.595+0000 7f072d1b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:09.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:09.611+0000 7f072d1b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:10.451 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:05:10.477 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:05:10.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:10.595+0000 7f0f75509640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:10.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:10.611+0000 7f0f75509640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:11.464 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:05:11.494 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:05:11.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:11.607+0000 7faa714bf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:11.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:11.623+0000 7faa714bf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:12.484 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:05:12.508 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:05:12.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:12.619+0000 7f7cc5759640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:12.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:12.635+0000 7f7cc5759640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:13.501 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:05:13.527 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:05:13.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:13.647+0000 7fbdb2cfe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:13.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:13.663+0000 7fbdb2cfe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:14.504 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:05:14.535 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:05:14.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:14.655+0000 7f24c4e0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:14.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:14.671+0000 7f24c4e0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:15.527 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:05:15.556 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:05:15.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:15.675+0000 7f2dcc0c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:15.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:15.691+0000 7f2dcc0c5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:16.540 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:05:16.567 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:05:16.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:16.687+0000 7f2458a72640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:16.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:16.703+0000 7f2458a72640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:17.559 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:05:17.587 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:05:17.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:17.707+0000 7f181f7ad640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:17.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:17.723+0000 7f181f7ad640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:18.569 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:05:18.594 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:05:18.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:18.707+0000 7fbe28881640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:18.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:18.723+0000 7fbe28881640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:19.574 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:05:19.601 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:05:19.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:19.718+0000 7fd7ca838640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:19.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:19.734+0000 7fd7ca838640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:05:20.611 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:05:20.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:20.730+0000 7f4f8470f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:20.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:20.746+0000 7f4f8470f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:21.602 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:05:21.629 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:05:21.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:21.746+0000 7efcf726c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:21.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:21.762+0000 7efcf726c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:22.617 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:05:22.643 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:05:22.643 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:05:22.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:22.762+0000 7ff64d46f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:22.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:22.778+0000 7ff64d46f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:22.955 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:05:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:05:22.975 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":554,"stamp":"2024-03-30T13:05:22.615798+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":208605646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":368832,"kb_used_data":118736,"kb_used_omap":34,"kb_used_meta":249949,"kb_avail":754605888,"statfs":{"total":773094113280,"available":772716429312,"internally_reserved":0,"allocated":121585664,"data_stored":107597295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35021,"internal_metadata":255948595},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.049554"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":560,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-30T13:05:16.538692+0000","last_change":"2024-03-30T13:05:07.445803+0000","last_active":"2024-03-30T13:05:16.538692+0000","last_peered":"2024-03-30T13:05:16.538692+0000","last_clean":"2024-03-30T13:05:16.538692+0000","last_became_active":"2024-03-30T13:05:07.444573+0000","last_became_peered":"2024-03-30T13:05:07.444573+0000","last_unstale":"2024-03-30T13:05:16.538692+0000","last_undegraded":"2024-03-30T13:05:16.538692+0000","last_fullsized":"2024-03-30T13:05:16.538692+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:04:16.526295+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:25.935052+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":535,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-30T13:05:20.671244+0000","last_change":"2024-03-30T13:05:09.190113+0000","last_active":"2024-03-30T13:05:20.671244+0000","last_peered":"2024-03-30T13:05:20.671244+0000","last_clean":"2024-03-30T13:05:20.671244+0000","last_became_active":"2024-03-30T13:05:07.444962+0000","last_became_peered":"2024-03-30T13:05:07.444962+0000","last_unstale":"2024-03-30T13:05:20.671244+0000","last_undegraded":"2024-03-30T13:05:20.671244+0000","last_fullsized":"2024-03-30T13:05:20.671244+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:05:09.189831+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:05:09.189831+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:14.013241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":165,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-30T13:05:16.993403+0000","last_change":"2024-03-30T13:05:07.444027+0000","last_active":"2024-03-30T13:05:16.993403+0000","last_peered":"2024-03-30T13:05:16.993403+0000","last_clean":"2024-03-30T13:05:16.993403+0000","last_became_active":"2024-03-30T13:05:07.442521+0000","last_became_peered":"2024-03-30T13:05:07.442521+0000","last_unstale":"2024-03-30T13:05:16.993403+0000","last_undegraded":"2024-03-30T13:05:16.993403+0000","last_fullsized":"2024-03-30T13:05:16.993403+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:04:55.327587+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_clean_scrub_stamp":"2024-03-30T13:04:55.327587+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:10.849965+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":162,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-30T13:05:16.993331+0000","last_change":"2024-03-30T13:04:36.430290+0000","last_active":"2024-03-30T13:05:16.993331+0000","last_peered":"2024-03-30T13:05:16.993331+0000","last_clean":"2024-03-30T13:05:16.993331+0000","last_became_active":"2024-03-30T13:04:16.039346+0000","last_became_peered":"2024-03-30T13:04:16.039346+0000","last_unstale":"2024-03-30T13:05:16.993331+0000","last_undegraded":"2024-03-30T13:05:16.993331+0000","last_fullsized":"2024-03-30T13:05:16.993331+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:36.430045+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:04:36.430045+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:04.050679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":241,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-30T13:05:20.671237+0000","last_change":"2024-03-30T13:05:17.370454+0000","last_active":"2024-03-30T13:05:20.671237+0000","last_peered":"2024-03-30T13:05:20.671237+0000","last_clean":"2024-03-30T13:05:20.671237+0000","last_became_active":"2024-03-30T13:01:41.984313+0000","last_became_peered":"2024-03-30T13:01:41.984313+0000","last_unstale":"2024-03-30T13:05:20.671237+0000","last_undegraded":"2024-03-30T13:05:20.671237+0000","last_fullsized":"2024-03-30T13:05:20.671237+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:05:17.370216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:05:17.370216+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:18.517958+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":592,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-30T13:05:16.994934+0000","last_change":"2024-03-30T13:05:13.431891+0000","last_active":"2024-03-30T13:05:16.994934+0000","last_peered":"2024-03-30T13:05:16.994934+0000","last_clean":"2024-03-30T13:05:16.994934+0000","last_became_active":"2024-03-30T13:01:41.987920+0000","last_became_peered":"2024-03-30T13:01:41.987920+0000","last_unstale":"2024-03-30T13:05:16.994934+0000","last_undegraded":"2024-03-30T13:05:16.994934+0000","last_fullsized":"2024-03-30T13:05:16.994934+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:05:13.431611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:05:13.431611+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:29.995673+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22789962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":190,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-30T13:05:16.993344+0000","last_change":"2024-03-30T13:05:07.441580+0000","last_active":"2024-03-30T13:05:16.993344+0000","last_peered":"2024-03-30T13:05:16.993344+0000","last_clean":"2024-03-30T13:05:16.993344+0000","last_became_active":"2024-03-30T13:05:07.440390+0000","last_became_peered":"2024-03-30T13:05:07.440390+0000","last_unstale":"2024-03-30T13:05:16.993344+0000","last_undegraded":"2024-03-30T13:05:16.993344+0000","last_fullsized":"2024-03-30T13:05:16.993344+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:04:27.400094+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:04:27.400094+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:51.289128+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":596,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-30T13:05:17.558229+0000","last_change":"2024-03-30T13:04:16.038106+0000","last_active":"2024-03-30T13:05:17.558229+0000","last_peered":"2024-03-30T13:05:17.558229+0000","last_clean":"2024-03-30T13:05:17.558229+0000","last_became_active":"2024-03-30T13:04:16.037485+0000","last_became_peered":"2024-03-30T13:04:16.037485+0000","last_unstale":"2024-03-30T13:05:17.558229+0000","last_undegraded":"2024-03-30T13:05:17.558229+0000","last_fullsized":"2024-03-30T13:05:17.558229+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:03:57.023686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:57.023686+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:20.958312+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29378240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":662,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-30T13:05:16.995099+0000","last_change":"2024-03-30T13:04:40.339554+0000","last_active":"2024-03-30T13:05:16.995099+0000","last_peered":"2024-03-30T13:05:16.995099+0000","last_clean":"2024-03-30T13:05:16.995099+0000","last_became_active":"2024-03-30T13:03:25.980366+0000","last_became_peered":"2024-03-30T13:03:25.980366+0000","last_unstale":"2024-03-30T13:05:16.995099+0000","last_undegraded":"2024-03-30T13:05:16.995099+0000","last_fullsized":"2024-03-30T13:05:16.995099+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:40.339270+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:40.339270+0000","last_clean_scrub_stamp":"2024-03-30T13:04:40.339270+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:05.690409+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":45680832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":461,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-30T13:05:20.670819+0000","last_change":"2024-03-30T13:05:12.144999+0000","last_active":"2024-03-30T13:05:20.670819+0000","last_peered":"2024-03-30T13:05:20.670819+0000","last_clean":"2024-03-30T13:05:20.670819+0000","last_became_active":"2024-03-30T13:05:07.445793+0000","last_became_peered":"2024-03-30T13:05:07.445793+0000","last_unstale":"2024-03-30T13:05:20.670819+0000","last_undegraded":"2024-03-30T13:05:20.670819+0000","last_fullsized":"2024-03-30T13:05:20.670819+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:05:12.144734+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:05:12.144734+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:30.214751+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":613,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-30T13:05:16.538509+0000","last_change":"2024-03-30T13:05:07.446715+0000","last_active":"2024-03-30T13:05:16.538509+0000","last_peered":"2024-03-30T13:05:16.538509+0000","last_clean":"2024-03-30T13:05:16.538509+0000","last_became_active":"2024-03-30T13:05:07.445475+0000","last_became_peered":"2024-03-30T13:05:07.445475+0000","last_unstale":"2024-03-30T13:05:16.538509+0000","last_undegraded":"2024-03-30T13:05:16.538509+0000","last_fullsized":"2024-03-30T13:05:16.538509+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:05:00.265024+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:05:00.265024+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:17.234152+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":586,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-30T13:05:16.538576+0000","last_change":"2024-03-30T13:04:53.373086+0000","last_active":"2024-03-30T13:05:16.538576+0000","last_peered":"2024-03-30T13:05:16.538576+0000","last_clean":"2024-03-30T13:05:16.538576+0000","last_became_active":"2024-03-30T13:04:16.049244+0000","last_became_peered":"2024-03-30T13:04:16.049244+0000","last_unstale":"2024-03-30T13:05:16.538576+0000","last_undegraded":"2024-03-30T13:05:16.538576+0000","last_fullsized":"2024-03-30T13:05:16.538576+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:04:53.372665+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:04:53.372665+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:05.305383+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":247,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-30T13:05:20.671206+0000","last_change":"2024-03-30T13:04:21.312886+0000","last_active":"2024-03-30T13:05:20.671206+0000","last_peered":"2024-03-30T13:05:20.671206+0000","last_clean":"2024-03-30T13:05:20.671206+0000","last_became_active":"2024-03-30T13:01:41.982566+0000","last_became_peered":"2024-03-30T13:01:41.982566+0000","last_unstale":"2024-03-30T13:05:20.671206+0000","last_undegraded":"2024-03-30T13:05:20.671206+0000","last_fullsized":"2024-03-30T13:05:20.671206+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_clean_scrub_stamp":"2024-03-30T13:04:21.312610+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:50.841034+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":142,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-30T13:05:16.993347+0000","last_change":"2024-03-30T13:04:51.422166+0000","last_active":"2024-03-30T13:05:16.993347+0000","last_peered":"2024-03-30T13:05:16.993347+0000","last_clean":"2024-03-30T13:05:16.993347+0000","last_became_active":"2024-03-30T13:03:25.178234+0000","last_became_peered":"2024-03-30T13:03:25.178234+0000","last_unstale":"2024-03-30T13:05:16.993347+0000","last_undegraded":"2024-03-30T13:05:16.993347+0000","last_fullsized":"2024-03-30T13:05:16.993347+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:04:51.421910+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:04:51.421910+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:54.940198+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":269,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-30T13:05:20.671270+0000","last_change":"2024-03-30T13:04:16.040294+0000","last_active":"2024-03-30T13:05:20.671270+0000","last_peered":"2024-03-30T13:05:20.671270+0000","last_clean":"2024-03-30T13:05:20.671270+0000","last_became_active":"2024-03-30T13:04:16.039183+0000","last_became_peered":"2024-03-30T13:04:16.039183+0000","last_unstale":"2024-03-30T13:05:20.671270+0000","last_undegraded":"2024-03-30T13:05:20.671270+0000","last_fullsized":"2024-03-30T13:05:20.671270+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:04:11.411056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:04:11.411056+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:21.985751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":569,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-30T13:05:17.558302+0000","last_change":"2024-03-30T13:04:55.119064+0000","last_active":"2024-03-30T13:05:17.558302+0000","last_peered":"2024-03-30T13:05:17.558302+0000","last_clean":"2024-03-30T13:05:17.558302+0000","last_became_active":"2024-03-30T13:01:41.986001+0000","last_became_peered":"2024-03-30T13:01:41.986001+0000","last_unstale":"2024-03-30T13:05:17.558302+0000","last_undegraded":"2024-03-30T13:05:17.558302+0000","last_fullsized":"2024-03-30T13:05:17.558302+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:55.118814+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:04:55.118814+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:17.644874+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10453822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":208605646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":227,"seq":974957576207,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49876,"kb_used_data":18180,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321964,"statfs":{"total":96636764160,"available":96585691136,"internally_reserved":0,"allocated":18616320,"data_stored":16875400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":156,"seq":670014898223,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54640,"kb_used_data":22948,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94317200,"statfs":{"total":96636764160,"available":96580812800,"internally_reserved":0,"allocated":23498752,"data_stored":21744280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4405,"internal_metadata":32435915},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824005,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47264,"kb_used_data":14676,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94324576,"statfs":{"total":96636764160,"available":96588365824,"internally_reserved":0,"allocated":15028224,"data_stored":13260150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509561,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40604,"kb_used_data":9420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331236,"statfs":{"total":96636764160,"available":96595185664,"internally_reserved":0,"allocated":9646080,"data_stored":7891608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285646,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39936,"kb_used_data":8240,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94331904,"statfs":{"total":96636764160,"available":96595869696,"internally_reserved":0,"allocated":8437760,"data_stored":6696840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509561,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47768,"kb_used_data":16584,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324072,"statfs":{"total":96636764160,"available":96587849728,"internally_reserved":0,"allocated":16982016,"data_stored":15234345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":206,"seq":884763263002,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46152,"kb_used_data":13116,"kb_used_omap":9,"kb_used_meta":33014,"kb_avail":94325688,"statfs":{"total":96636764160,"available":96589504512,"internally_reserved":0,"allocated":13430784,"data_stored":11689864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9338,"internal_metadata":33807238},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574969,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42592,"kb_used_data":15572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329248,"statfs":{"total":96636764160,"available":96593149952,"internally_reserved":0,"allocated":15945728,"data_stored":14204808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:05:22.977 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:05:23.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:23.098+0000 7f51ae9df640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:23.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:23.110+0000 7f51ae9df640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:23.285 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:05:23.285 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:05:23.302 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":554,"stamp":"2024-03-30T13:05:22.615798+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":208605646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":368832,"kb_used_data":118736,"kb_used_omap":34,"kb_used_meta":249949,"kb_avail":754605888,"statfs":{"total":773094113280,"available":772716429312,"internally_reserved":0,"allocated":121585664,"data_stored":107597295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35021,"internal_metadata":255948595},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.049554"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":560,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-30T13:05:16.538692+0000","last_change":"2024-03-30T13:05:07.445803+0000","last_active":"2024-03-30T13:05:16.538692+0000","last_peered":"2024-03-30T13:05:16.538692+0000","last_clean":"2024-03-30T13:05:16.538692+0000","last_became_active":"2024-03-30T13:05:07.444573+0000","last_became_peered":"2024-03-30T13:05:07.444573+0000","last_unstale":"2024-03-30T13:05:16.538692+0000","last_undegraded":"2024-03-30T13:05:16.538692+0000","last_fullsized":"2024-03-30T13:05:16.538692+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:04:16.526295+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:25.935052+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":535,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-30T13:05:20.671244+0000","last_change":"2024-03-30T13:05:09.190113+0000","last_active":"2024-03-30T13:05:20.671244+0000","last_peered":"2024-03-30T13:05:20.671244+0000","last_clean":"2024-03-30T13:05:20.671244+0000","last_became_active":"2024-03-30T13:05:07.444962+0000","last_became_peered":"2024-03-30T13:05:07.444962+0000","last_unstale":"2024-03-30T13:05:20.671244+0000","last_undegraded":"2024-03-30T13:05:20.671244+0000","last_fullsized":"2024-03-30T13:05:20.671244+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:05:09.189831+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:05:09.189831+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:14.013241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":165,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-30T13:05:16.993403+0000","last_change":"2024-03-30T13:05:07.444027+0000","last_active":"2024-03-30T13:05:16.993403+0000","last_peered":"2024-03-30T13:05:16.993403+0000","last_clean":"2024-03-30T13:05:16.993403+0000","last_became_active":"2024-03-30T13:05:07.442521+0000","last_became_peered":"2024-03-30T13:05:07.442521+0000","last_unstale":"2024-03-30T13:05:16.993403+0000","last_undegraded":"2024-03-30T13:05:16.993403+0000","last_fullsized":"2024-03-30T13:05:16.993403+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:04:55.327587+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_clean_scrub_stamp":"2024-03-30T13:04:55.327587+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:10.849965+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":162,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-30T13:05:16.993331+0000","last_change":"2024-03-30T13:04:36.430290+0000","last_active":"2024-03-30T13:05:16.993331+0000","last_peered":"2024-03-30T13:05:16.993331+0000","last_clean":"2024-03-30T13:05:16.993331+0000","last_became_active":"2024-03-30T13:04:16.039346+0000","last_became_peered":"2024-03-30T13:04:16.039346+0000","last_unstale":"2024-03-30T13:05:16.993331+0000","last_undegraded":"2024-03-30T13:05:16.993331+0000","last_fullsized":"2024-03-30T13:05:16.993331+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:36.430045+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:04:36.430045+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:04.050679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":241,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-30T13:05:20.671237+0000","last_change":"2024-03-30T13:05:17.370454+0000","last_active":"2024-03-30T13:05:20.671237+0000","last_peered":"2024-03-30T13:05:20.671237+0000","last_clean":"2024-03-30T13:05:20.671237+0000","last_became_active":"2024-03-30T13:01:41.984313+0000","last_became_peered":"2024-03-30T13:01:41.984313+0000","last_unstale":"2024-03-30T13:05:20.671237+0000","last_undegraded":"2024-03-30T13:05:20.671237+0000","last_fullsized":"2024-03-30T13:05:20.671237+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:05:17.370216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:05:17.370216+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:18.517958+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":592,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-30T13:05:16.994934+0000","last_change":"2024-03-30T13:05:13.431891+0000","last_active":"2024-03-30T13:05:16.994934+0000","last_peered":"2024-03-30T13:05:16.994934+0000","last_clean":"2024-03-30T13:05:16.994934+0000","last_became_active":"2024-03-30T13:01:41.987920+0000","last_became_peered":"2024-03-30T13:01:41.987920+0000","last_unstale":"2024-03-30T13:05:16.994934+0000","last_undegraded":"2024-03-30T13:05:16.994934+0000","last_fullsized":"2024-03-30T13:05:16.994934+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:05:13.431611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:05:13.431611+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:29.995673+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22789962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":190,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-30T13:05:16.993344+0000","last_change":"2024-03-30T13:05:07.441580+0000","last_active":"2024-03-30T13:05:16.993344+0000","last_peered":"2024-03-30T13:05:16.993344+0000","last_clean":"2024-03-30T13:05:16.993344+0000","last_became_active":"2024-03-30T13:05:07.440390+0000","last_became_peered":"2024-03-30T13:05:07.440390+0000","last_unstale":"2024-03-30T13:05:16.993344+0000","last_undegraded":"2024-03-30T13:05:16.993344+0000","last_fullsized":"2024-03-30T13:05:16.993344+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:04:27.400094+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:04:27.400094+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:51.289128+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":596,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-30T13:05:17.558229+0000","last_change":"2024-03-30T13:04:16.038106+0000","last_active":"2024-03-30T13:05:17.558229+0000","last_peered":"2024-03-30T13:05:17.558229+0000","last_clean":"2024-03-30T13:05:17.558229+0000","last_became_active":"2024-03-30T13:04:16.037485+0000","last_became_peered":"2024-03-30T13:04:16.037485+0000","last_unstale":"2024-03-30T13:05:17.558229+0000","last_undegraded":"2024-03-30T13:05:17.558229+0000","last_fullsized":"2024-03-30T13:05:17.558229+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:03:57.023686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:57.023686+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:20.958312+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29378240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":662,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-30T13:05:16.995099+0000","last_change":"2024-03-30T13:04:40.339554+0000","last_active":"2024-03-30T13:05:16.995099+0000","last_peered":"2024-03-30T13:05:16.995099+0000","last_clean":"2024-03-30T13:05:16.995099+0000","last_became_active":"2024-03-30T13:03:25.980366+0000","last_became_peered":"2024-03-30T13:03:25.980366+0000","last_unstale":"2024-03-30T13:05:16.995099+0000","last_undegraded":"2024-03-30T13:05:16.995099+0000","last_fullsized":"2024-03-30T13:05:16.995099+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:40.339270+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:40.339270+0000","last_clean_scrub_stamp":"2024-03-30T13:04:40.339270+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:05.690409+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":45680832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":461,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-30T13:05:20.670819+0000","last_change":"2024-03-30T13:05:12.144999+0000","last_active":"2024-03-30T13:05:20.670819+0000","last_peered":"2024-03-30T13:05:20.670819+0000","last_clean":"2024-03-30T13:05:20.670819+0000","last_became_active":"2024-03-30T13:05:07.445793+0000","last_became_peered":"2024-03-30T13:05:07.445793+0000","last_unstale":"2024-03-30T13:05:20.670819+0000","last_undegraded":"2024-03-30T13:05:20.670819+0000","last_fullsized":"2024-03-30T13:05:20.670819+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:05:12.144734+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:05:12.144734+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:30.214751+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":613,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-30T13:05:16.538509+0000","last_change":"2024-03-30T13:05:07.446715+0000","last_active":"2024-03-30T13:05:16.538509+0000","last_peered":"2024-03-30T13:05:16.538509+0000","last_clean":"2024-03-30T13:05:16.538509+0000","last_became_active":"2024-03-30T13:05:07.445475+0000","last_became_peered":"2024-03-30T13:05:07.445475+0000","last_unstale":"2024-03-30T13:05:16.538509+0000","last_undegraded":"2024-03-30T13:05:16.538509+0000","last_fullsized":"2024-03-30T13:05:16.538509+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:05:00.265024+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:05:00.265024+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:17.234152+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":586,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-30T13:05:16.538576+0000","last_change":"2024-03-30T13:04:53.373086+0000","last_active":"2024-03-30T13:05:16.538576+0000","last_peered":"2024-03-30T13:05:16.538576+0000","last_clean":"2024-03-30T13:05:16.538576+0000","last_became_active":"2024-03-30T13:04:16.049244+0000","last_became_peered":"2024-03-30T13:04:16.049244+0000","last_unstale":"2024-03-30T13:05:16.538576+0000","last_undegraded":"2024-03-30T13:05:16.538576+0000","last_fullsized":"2024-03-30T13:05:16.538576+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:04:53.372665+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:04:53.372665+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:05.305383+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":247,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-30T13:05:20.671206+0000","last_change":"2024-03-30T13:04:21.312886+0000","last_active":"2024-03-30T13:05:20.671206+0000","last_peered":"2024-03-30T13:05:20.671206+0000","last_clean":"2024-03-30T13:05:20.671206+0000","last_became_active":"2024-03-30T13:01:41.982566+0000","last_became_peered":"2024-03-30T13:01:41.982566+0000","last_unstale":"2024-03-30T13:05:20.671206+0000","last_undegraded":"2024-03-30T13:05:20.671206+0000","last_fullsized":"2024-03-30T13:05:20.671206+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_clean_scrub_stamp":"2024-03-30T13:04:21.312610+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:50.841034+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":142,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-30T13:05:16.993347+0000","last_change":"2024-03-30T13:04:51.422166+0000","last_active":"2024-03-30T13:05:16.993347+0000","last_peered":"2024-03-30T13:05:16.993347+0000","last_clean":"2024-03-30T13:05:16.993347+0000","last_became_active":"2024-03-30T13:03:25.178234+0000","last_became_peered":"2024-03-30T13:03:25.178234+0000","last_unstale":"2024-03-30T13:05:16.993347+0000","last_undegraded":"2024-03-30T13:05:16.993347+0000","last_fullsized":"2024-03-30T13:05:16.993347+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:04:51.421910+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:04:51.421910+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:54.940198+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":269,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-30T13:05:20.671270+0000","last_change":"2024-03-30T13:04:16.040294+0000","last_active":"2024-03-30T13:05:20.671270+0000","last_peered":"2024-03-30T13:05:20.671270+0000","last_clean":"2024-03-30T13:05:20.671270+0000","last_became_active":"2024-03-30T13:04:16.039183+0000","last_became_peered":"2024-03-30T13:04:16.039183+0000","last_unstale":"2024-03-30T13:05:20.671270+0000","last_undegraded":"2024-03-30T13:05:20.671270+0000","last_fullsized":"2024-03-30T13:05:20.671270+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:04:11.411056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:04:11.411056+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:21.985751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":569,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-03-30T13:05:17.558302+0000","last_change":"2024-03-30T13:04:55.119064+0000","last_active":"2024-03-30T13:05:17.558302+0000","last_peered":"2024-03-30T13:05:17.558302+0000","last_clean":"2024-03-30T13:05:17.558302+0000","last_became_active":"2024-03-30T13:01:41.986001+0000","last_became_peered":"2024-03-30T13:01:41.986001+0000","last_unstale":"2024-03-30T13:05:17.558302+0000","last_undegraded":"2024-03-30T13:05:17.558302+0000","last_fullsized":"2024-03-30T13:05:17.558302+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:55.118814+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:04:55.118814+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:17.644874+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10453822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":208605646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":227,"seq":974957576207,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49876,"kb_used_data":18180,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321964,"statfs":{"total":96636764160,"available":96585691136,"internally_reserved":0,"allocated":18616320,"data_stored":16875400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":156,"seq":670014898223,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54640,"kb_used_data":22948,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94317200,"statfs":{"total":96636764160,"available":96580812800,"internally_reserved":0,"allocated":23498752,"data_stored":21744280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4405,"internal_metadata":32435915},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824005,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47264,"kb_used_data":14676,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94324576,"statfs":{"total":96636764160,"available":96588365824,"internally_reserved":0,"allocated":15028224,"data_stored":13260150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509561,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40604,"kb_used_data":9420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331236,"statfs":{"total":96636764160,"available":96595185664,"internally_reserved":0,"allocated":9646080,"data_stored":7891608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285646,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39936,"kb_used_data":8240,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94331904,"statfs":{"total":96636764160,"available":96595869696,"internally_reserved":0,"allocated":8437760,"data_stored":6696840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509561,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47768,"kb_used_data":16584,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324072,"statfs":{"total":96636764160,"available":96587849728,"internally_reserved":0,"allocated":16982016,"data_stored":15234345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":206,"seq":884763263002,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46152,"kb_used_data":13116,"kb_used_omap":9,"kb_used_meta":33014,"kb_avail":94325688,"statfs":{"total":96636764160,"available":96589504512,"internally_reserved":0,"allocated":13430784,"data_stored":11689864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9338,"internal_metadata":33807238},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574969,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42592,"kb_used_data":15572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329248,"statfs":{"total":96636764160,"available":96593149952,"internally_reserved":0,"allocated":15945728,"data_stored":14204808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:05:23.304 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:05:23.304 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:05:23.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:23.406+0000 7fc764cab640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:23.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:23.422+0000 7fc764cab640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:23.591 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:05:23.592 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":266,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:05:22.605694+0000","last_up_change":"2024-03-30T13:05:06.405280+0000","last_in_change":"2024-03-30T13:05:06.405280+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":16,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":77,"up_from":83,"up_thru":227,"down_at":78,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6818","nonce":3104224893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6821","nonce":3104224893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6828","nonce":3104224893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6825","nonce":3104224893}]},"public_addr":"172.21.15.67:6818/3104224893","cluster_addr":"172.21.15.67:6821/3104224893","heartbeat_back_addr":"172.21.15.67:6828/3104224893","heartbeat_front_addr":"172.21.15.67:6825/3104224893","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":223,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6801","nonce":1504348724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6803","nonce":1504348724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6807","nonce":1504348724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6805","nonce":1504348724}]},"public_addr":"172.21.15.67:6801/1504348724","cluster_addr":"172.21.15.67:6803/1504348724","heartbeat_back_addr":"172.21.15.67:6807/1504348724","heartbeat_front_addr":"172.21.15.67:6805/1504348724","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":135,"last_clean_end":243,"up_from":250,"up_thru":250,"down_at":244,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6803","nonce":3986522908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6807","nonce":3986522908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6815","nonce":3986522908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6811","nonce":3986522908}]},"public_addr":"172.21.15.82:6803/3986522908","cluster_addr":"172.21.15.82:6807/3986522908","heartbeat_back_addr":"172.21.15.82:6815/3986522908","heartbeat_front_addr":"172.21.15.82:6811/3986522908","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":14,"last_clean_end":151,"up_from":156,"up_thru":227,"down_at":152,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":3428934162},{"type":"v1","addr":"172.21.15.82:6817","nonce":3428934162}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":3428934162},{"type":"v1","addr":"172.21.15.82:6819","nonce":3428934162}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":3428934162},{"type":"v1","addr":"172.21.15.82:6823","nonce":3428934162}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":3428934162},{"type":"v1","addr":"172.21.15.82:6821","nonce":3428934162}]},"public_addr":"172.21.15.82:6817/3428934162","cluster_addr":"172.21.15.82:6819/3428934162","heartbeat_back_addr":"172.21.15.82:6823/3428934162","heartbeat_front_addr":"172.21.15.82:6821/3428934162","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":222,"up_from":227,"up_thru":250,"down_at":223,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3068584906},{"type":"v1","addr":"172.21.15.82:6801","nonce":3068584906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3068584906},{"type":"v1","addr":"172.21.15.82:6806","nonce":3068584906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3068584906},{"type":"v1","addr":"172.21.15.82:6814","nonce":3068584906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3068584906},{"type":"v1","addr":"172.21.15.82:6810","nonce":3068584906}]},"public_addr":"172.21.15.82:6801/3068584906","cluster_addr":"172.21.15.82:6806/3068584906","heartbeat_back_addr":"172.21.15.82:6814/3068584906","heartbeat_front_addr":"172.21.15.82:6810/3068584906","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":180,"last_clean_end":197,"up_from":206,"up_thru":250,"down_at":198,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":1241244807},{"type":"v1","addr":"172.21.15.82:6825","nonce":1241244807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":1241244807},{"type":"v1","addr":"172.21.15.82:6827","nonce":1241244807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":1241244807},{"type":"v1","addr":"172.21.15.82:6831","nonce":1241244807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":1241244807},{"type":"v1","addr":"172.21.15.82:6829","nonce":1241244807}]},"public_addr":"172.21.15.82:6825/1241244807","cluster_addr":"172.21.15.82:6827/1241244807","heartbeat_back_addr":"172.21.15.82:6831/1241244807","heartbeat_front_addr":"172.21.15.82:6829/1241244807","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:04:26.479183+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:04:37.476063+0000","dead_epoch":78},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:04:15.285792+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:04:39.879994+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:04:26.573314+0000","dead_epoch":244},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:04:33.149109+0000","dead_epoch":152},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:04:14.477530+0000","dead_epoch":223},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:05:16.544575+0000","dead_epoch":198}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:05:23.609 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:05:23.609 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:05:23.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:23.722+0000 7f9d113b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:23.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:23.738+0000 7f9d113b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:23.906 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:05:23.924 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:05:24.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:24.038+0000 7fadd5693640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:24.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:24.054+0000 7fadd5693640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:24.229 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:05:24.230 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:05:24.248 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:05:24.249 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:05:24.249 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:05:24.249 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:05:24.249 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 3, 1, 5, 7, 6, 4] 2024-03-30T13:05:24.249 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:05:24.250 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:05:24.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:24.350+0000 7f5801dc6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:24.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:24.366+0000 7f5801dc6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:24.532 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:05:24.533 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:05:24.549 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":555,"stamp":"2024-03-30T13:05:23.950403+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":208605646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":368964,"kb_used_data":118856,"kb_used_omap":34,"kb_used_meta":249949,"kb_avail":754605756,"statfs":{"total":773094113280,"available":772716294144,"internally_reserved":0,"allocated":121708544,"data_stored":107652265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35021,"internal_metadata":255948595},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.373610"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":560,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-30T13:05:16.538692+0000","last_change":"2024-03-30T13:05:07.445803+0000","last_active":"2024-03-30T13:05:16.538692+0000","last_peered":"2024-03-30T13:05:16.538692+0000","last_clean":"2024-03-30T13:05:16.538692+0000","last_became_active":"2024-03-30T13:05:07.444573+0000","last_became_peered":"2024-03-30T13:05:07.444573+0000","last_unstale":"2024-03-30T13:05:16.538692+0000","last_undegraded":"2024-03-30T13:05:16.538692+0000","last_fullsized":"2024-03-30T13:05:16.538692+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:04:16.526295+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:25.935052+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":535,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-30T13:05:20.671244+0000","last_change":"2024-03-30T13:05:09.190113+0000","last_active":"2024-03-30T13:05:20.671244+0000","last_peered":"2024-03-30T13:05:20.671244+0000","last_clean":"2024-03-30T13:05:20.671244+0000","last_became_active":"2024-03-30T13:05:07.444962+0000","last_became_peered":"2024-03-30T13:05:07.444962+0000","last_unstale":"2024-03-30T13:05:20.671244+0000","last_undegraded":"2024-03-30T13:05:20.671244+0000","last_fullsized":"2024-03-30T13:05:20.671244+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:05:09.189831+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:05:09.189831+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:14.013241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":165,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-30T13:05:16.993403+0000","last_change":"2024-03-30T13:05:07.444027+0000","last_active":"2024-03-30T13:05:16.993403+0000","last_peered":"2024-03-30T13:05:16.993403+0000","last_clean":"2024-03-30T13:05:16.993403+0000","last_became_active":"2024-03-30T13:05:07.442521+0000","last_became_peered":"2024-03-30T13:05:07.442521+0000","last_unstale":"2024-03-30T13:05:16.993403+0000","last_undegraded":"2024-03-30T13:05:16.993403+0000","last_fullsized":"2024-03-30T13:05:16.993403+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:04:55.327587+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_clean_scrub_stamp":"2024-03-30T13:04:55.327587+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:10.849965+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":162,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-30T13:05:16.993331+0000","last_change":"2024-03-30T13:04:36.430290+0000","last_active":"2024-03-30T13:05:16.993331+0000","last_peered":"2024-03-30T13:05:16.993331+0000","last_clean":"2024-03-30T13:05:16.993331+0000","last_became_active":"2024-03-30T13:04:16.039346+0000","last_became_peered":"2024-03-30T13:04:16.039346+0000","last_unstale":"2024-03-30T13:05:16.993331+0000","last_undegraded":"2024-03-30T13:05:16.993331+0000","last_fullsized":"2024-03-30T13:05:16.993331+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:36.430045+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:04:36.430045+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:04.050679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":241,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-30T13:05:20.671237+0000","last_change":"2024-03-30T13:05:17.370454+0000","last_active":"2024-03-30T13:05:20.671237+0000","last_peered":"2024-03-30T13:05:20.671237+0000","last_clean":"2024-03-30T13:05:20.671237+0000","last_became_active":"2024-03-30T13:01:41.984313+0000","last_became_peered":"2024-03-30T13:01:41.984313+0000","last_unstale":"2024-03-30T13:05:20.671237+0000","last_undegraded":"2024-03-30T13:05:20.671237+0000","last_fullsized":"2024-03-30T13:05:20.671237+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:05:17.370216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:05:17.370216+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:18.517958+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":602,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-30T13:05:22.789864+0000","last_change":"2024-03-30T13:05:13.431891+0000","last_active":"2024-03-30T13:05:22.789864+0000","last_peered":"2024-03-30T13:05:22.789864+0000","last_clean":"2024-03-30T13:05:22.789864+0000","last_became_active":"2024-03-30T13:01:41.987920+0000","last_became_peered":"2024-03-30T13:01:41.987920+0000","last_unstale":"2024-03-30T13:05:22.789864+0000","last_undegraded":"2024-03-30T13:05:22.789864+0000","last_fullsized":"2024-03-30T13:05:22.789864+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:05:13.431611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:05:13.431611+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:29.995673+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22789962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":190,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-30T13:05:16.993344+0000","last_change":"2024-03-30T13:05:07.441580+0000","last_active":"2024-03-30T13:05:16.993344+0000","last_peered":"2024-03-30T13:05:16.993344+0000","last_clean":"2024-03-30T13:05:16.993344+0000","last_became_active":"2024-03-30T13:05:07.440390+0000","last_became_peered":"2024-03-30T13:05:07.440390+0000","last_unstale":"2024-03-30T13:05:16.993344+0000","last_undegraded":"2024-03-30T13:05:16.993344+0000","last_fullsized":"2024-03-30T13:05:16.993344+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:04:27.400094+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:04:27.400094+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:51.289128+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":606,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-30T13:05:22.789283+0000","last_change":"2024-03-30T13:04:16.038106+0000","last_active":"2024-03-30T13:05:22.789283+0000","last_peered":"2024-03-30T13:05:22.789283+0000","last_clean":"2024-03-30T13:05:22.789283+0000","last_became_active":"2024-03-30T13:04:16.037485+0000","last_became_peered":"2024-03-30T13:04:16.037485+0000","last_unstale":"2024-03-30T13:05:22.789283+0000","last_undegraded":"2024-03-30T13:05:22.789283+0000","last_fullsized":"2024-03-30T13:05:22.789283+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:03:57.023686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:57.023686+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29378240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":672,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-30T13:05:22.789900+0000","last_change":"2024-03-30T13:04:40.339554+0000","last_active":"2024-03-30T13:05:22.789900+0000","last_peered":"2024-03-30T13:05:22.789900+0000","last_clean":"2024-03-30T13:05:22.789900+0000","last_became_active":"2024-03-30T13:03:25.980366+0000","last_became_peered":"2024-03-30T13:03:25.980366+0000","last_unstale":"2024-03-30T13:05:22.789900+0000","last_undegraded":"2024-03-30T13:05:22.789900+0000","last_fullsized":"2024-03-30T13:05:22.789900+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:40.339270+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:40.339270+0000","last_clean_scrub_stamp":"2024-03-30T13:04:40.339270+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:05.690409+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":45680832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":461,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-30T13:05:20.670819+0000","last_change":"2024-03-30T13:05:12.144999+0000","last_active":"2024-03-30T13:05:20.670819+0000","last_peered":"2024-03-30T13:05:20.670819+0000","last_clean":"2024-03-30T13:05:20.670819+0000","last_became_active":"2024-03-30T13:05:07.445793+0000","last_became_peered":"2024-03-30T13:05:07.445793+0000","last_unstale":"2024-03-30T13:05:20.670819+0000","last_undegraded":"2024-03-30T13:05:20.670819+0000","last_fullsized":"2024-03-30T13:05:20.670819+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:05:12.144734+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:05:12.144734+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:30.214751+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":613,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-30T13:05:16.538509+0000","last_change":"2024-03-30T13:05:07.446715+0000","last_active":"2024-03-30T13:05:16.538509+0000","last_peered":"2024-03-30T13:05:16.538509+0000","last_clean":"2024-03-30T13:05:16.538509+0000","last_became_active":"2024-03-30T13:05:07.445475+0000","last_became_peered":"2024-03-30T13:05:07.445475+0000","last_unstale":"2024-03-30T13:05:16.538509+0000","last_undegraded":"2024-03-30T13:05:16.538509+0000","last_fullsized":"2024-03-30T13:05:16.538509+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:05:00.265024+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:05:00.265024+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:17.234152+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":586,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-30T13:05:16.538576+0000","last_change":"2024-03-30T13:04:53.373086+0000","last_active":"2024-03-30T13:05:16.538576+0000","last_peered":"2024-03-30T13:05:16.538576+0000","last_clean":"2024-03-30T13:05:16.538576+0000","last_became_active":"2024-03-30T13:04:16.049244+0000","last_became_peered":"2024-03-30T13:04:16.049244+0000","last_unstale":"2024-03-30T13:05:16.538576+0000","last_undegraded":"2024-03-30T13:05:16.538576+0000","last_fullsized":"2024-03-30T13:05:16.538576+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:04:53.372665+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:04:53.372665+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:05.305383+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":247,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-30T13:05:20.671206+0000","last_change":"2024-03-30T13:04:21.312886+0000","last_active":"2024-03-30T13:05:20.671206+0000","last_peered":"2024-03-30T13:05:20.671206+0000","last_clean":"2024-03-30T13:05:20.671206+0000","last_became_active":"2024-03-30T13:01:41.982566+0000","last_became_peered":"2024-03-30T13:01:41.982566+0000","last_unstale":"2024-03-30T13:05:20.671206+0000","last_undegraded":"2024-03-30T13:05:20.671206+0000","last_fullsized":"2024-03-30T13:05:20.671206+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_clean_scrub_stamp":"2024-03-30T13:04:21.312610+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:50.841034+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":142,"reported_epoch":260,"state":"active+clean","last_fresh":"2024-03-30T13:05:16.993347+0000","last_change":"2024-03-30T13:04:51.422166+0000","last_active":"2024-03-30T13:05:16.993347+0000","last_peered":"2024-03-30T13:05:16.993347+0000","last_clean":"2024-03-30T13:05:16.993347+0000","last_became_active":"2024-03-30T13:03:25.178234+0000","last_became_peered":"2024-03-30T13:03:25.178234+0000","last_unstale":"2024-03-30T13:05:16.993347+0000","last_undegraded":"2024-03-30T13:05:16.993347+0000","last_fullsized":"2024-03-30T13:05:16.993347+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:04:51.421910+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:04:51.421910+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:54.940198+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":269,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-30T13:05:20.671270+0000","last_change":"2024-03-30T13:04:16.040294+0000","last_active":"2024-03-30T13:05:20.671270+0000","last_peered":"2024-03-30T13:05:20.671270+0000","last_clean":"2024-03-30T13:05:20.671270+0000","last_became_active":"2024-03-30T13:04:16.039183+0000","last_became_peered":"2024-03-30T13:04:16.039183+0000","last_unstale":"2024-03-30T13:05:20.671270+0000","last_undegraded":"2024-03-30T13:05:20.671270+0000","last_fullsized":"2024-03-30T13:05:20.671270+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:04:11.411056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:04:11.411056+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:21.985751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":579,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-30T13:05:22.789284+0000","last_change":"2024-03-30T13:04:55.119064+0000","last_active":"2024-03-30T13:05:22.789284+0000","last_peered":"2024-03-30T13:05:22.789284+0000","last_clean":"2024-03-30T13:05:22.789284+0000","last_became_active":"2024-03-30T13:01:41.986001+0000","last_became_peered":"2024-03-30T13:01:41.986001+0000","last_unstale":"2024-03-30T13:05:22.789284+0000","last_undegraded":"2024-03-30T13:05:22.789284+0000","last_fullsized":"2024-03-30T13:05:22.789284+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:55.118814+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:04:55.118814+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:17.644874+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10453822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":208605646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":227,"seq":974957576207,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49876,"kb_used_data":18180,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321964,"statfs":{"total":96636764160,"available":96585691136,"internally_reserved":0,"allocated":18616320,"data_stored":16875400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":156,"seq":670014898223,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54640,"kb_used_data":22948,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94317200,"statfs":{"total":96636764160,"available":96580812800,"internally_reserved":0,"allocated":23498752,"data_stored":21744280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4405,"internal_metadata":32435915},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824005,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47264,"kb_used_data":14676,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94324576,"statfs":{"total":96636764160,"available":96588365824,"internally_reserved":0,"allocated":15028224,"data_stored":13260150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509561,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40604,"kb_used_data":9420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331236,"statfs":{"total":96636764160,"available":96595185664,"internally_reserved":0,"allocated":9646080,"data_stored":7891608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285647,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40000,"kb_used_data":8300,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94331840,"statfs":{"total":96636764160,"available":96595804160,"internally_reserved":0,"allocated":8499200,"data_stored":6724325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509561,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47768,"kb_used_data":16584,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324072,"statfs":{"total":96636764160,"available":96587849728,"internally_reserved":0,"allocated":16982016,"data_stored":15234345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":206,"seq":884763263002,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46152,"kb_used_data":13116,"kb_used_omap":9,"kb_used_meta":33014,"kb_avail":94325688,"statfs":{"total":96636764160,"available":96589504512,"internally_reserved":0,"allocated":13430784,"data_stored":11689864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9338,"internal_metadata":33807238},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574970,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42660,"kb_used_data":15632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329180,"statfs":{"total":96636764160,"available":96593080320,"internally_reserved":0,"allocated":16007168,"data_stored":14232293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:05:24.550 INFO:tasks.thrashosds.thrasher:Choosing PG 1.1 with acting set [5, 2, 0] 2024-03-30T13:05:24.550 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 2, 3, 1, 5, 7, 6, 4] 2024-03-30T13:05:24.550 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-30T13:05:24.550 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:05:24.608 INFO:tasks.ceph.osd.5:Stopped 2024-03-30T13:05:24.608 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 2, 3, 1, 5, 7, 6, 4] 2024-03-30T13:05:24.608 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 5 2024-03-30T13:05:24.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:24.726+0000 7fc4ea447640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:24.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:24.742+0000 7fc4ea447640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:24.922 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:05:24.918+0000 7fbb38f08640 -1 mon.a@0(leader).osd e267 definitely_dead 0 2024-03-30T13:05:25.649 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.5. 2024-03-30T13:05:25.676 INFO:tasks.thrashosds.thrasher:dead_osds=[5], live_osds=[0, 2, 3, 1, 7, 6, 4] 2024-03-30T13:05:25.676 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:05:25.676 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:05:25.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:25.794+0000 7f4006807640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:25.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:25.810+0000 7f4006807640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:25.985 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:05:25.986 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:05:26.003 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":558,"stamp":"2024-03-30T13:05:25.950973+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":179059369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":314600,"kb_used_data":96160,"kb_used_omap":29,"kb_used_meta":218274,"kb_avail":660288280,"statfs":{"total":676457349120,"available":676135198720,"internally_reserved":0,"allocated":98467840,"data_stored":86023312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30616,"internal_metadata":223512680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-13,"num_bytes_recovered":-29546277,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.345317"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":571,"reported_epoch":266,"state":"peering","last_fresh":"2024-03-30T13:05:22.618379+0000","last_change":"2024-03-30T13:05:24.628882+0000","last_active":"2024-03-30T13:05:24.628882+0000","last_peered":"2024-03-30T13:05:22.618379+0000","last_clean":"2024-03-30T13:05:22.618379+0000","last_became_active":"2024-03-30T13:05:07.444573+0000","last_became_peered":"2024-03-30T13:05:07.444573+0000","last_unstale":"2024-03-30T13:05:22.618379+0000","last_undegraded":"2024-03-30T13:05:22.618379+0000","last_fullsized":"2024-03-30T13:05:22.618379+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:04:16.526295+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2147483647,4],"acting":[6,2147483647,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":535,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-30T13:05:20.671244+0000","last_change":"2024-03-30T13:05:09.190113+0000","last_active":"2024-03-30T13:05:20.671244+0000","last_peered":"2024-03-30T13:05:20.671244+0000","last_clean":"2024-03-30T13:05:20.671244+0000","last_became_active":"2024-03-30T13:05:07.444962+0000","last_became_peered":"2024-03-30T13:05:07.444962+0000","last_unstale":"2024-03-30T13:05:20.671244+0000","last_undegraded":"2024-03-30T13:05:20.671244+0000","last_fullsized":"2024-03-30T13:05:20.671244+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:05:09.189831+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:05:09.189831+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:14.013241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":175,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-30T13:05:22.790207+0000","last_change":"2024-03-30T13:05:07.444027+0000","last_active":"2024-03-30T13:05:22.790207+0000","last_peered":"2024-03-30T13:05:22.790207+0000","last_clean":"2024-03-30T13:05:22.790207+0000","last_became_active":"2024-03-30T13:05:07.442521+0000","last_became_peered":"2024-03-30T13:05:07.442521+0000","last_unstale":"2024-03-30T13:05:22.790207+0000","last_undegraded":"2024-03-30T13:05:22.790207+0000","last_fullsized":"2024-03-30T13:05:22.790207+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:04:55.327587+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_clean_scrub_stamp":"2024-03-30T13:04:55.327587+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:10.849965+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":173,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-30T13:05:22.790449+0000","last_change":"2024-03-30T13:04:36.430290+0000","last_active":"2024-03-30T13:05:22.790449+0000","last_peered":"2024-03-30T13:05:22.790449+0000","last_clean":"2024-03-30T13:05:22.790449+0000","last_became_active":"2024-03-30T13:04:16.039346+0000","last_became_peered":"2024-03-30T13:04:16.039346+0000","last_unstale":"2024-03-30T13:05:22.790449+0000","last_undegraded":"2024-03-30T13:05:22.790449+0000","last_fullsized":"2024-03-30T13:05:22.790449+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:36.430045+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:04:36.430045+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:04.050679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":22,"reported_epoch":267,"state":"peering","last_fresh":"2024-03-30T13:05:24.629507+0000","last_change":"2024-03-30T13:01:03.279745+0000","last_active":"2024-03-30T13:01:03.279745+0000","last_peered":"2024-03-30T12:55:47.164508+0000","last_clean":"2024-03-30T12:55:47.164508+0000","last_became_active":"2024-03-30T12:55:43.933770+0000","last_became_peered":"2024-03-30T12:55:43.933770+0000","last_unstale":"2024-03-30T13:05:24.629507+0000","last_undegraded":"2024-03-30T13:05:24.629507+0000","last_fullsized":"2024-03-30T13:05:24.629507+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:05:17.370216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:05:17.370216+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,0],"acting":[2147483647,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":602,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-30T13:05:22.789864+0000","last_change":"2024-03-30T13:05:13.431891+0000","last_active":"2024-03-30T13:05:22.789864+0000","last_peered":"2024-03-30T13:05:22.789864+0000","last_clean":"2024-03-30T13:05:22.789864+0000","last_became_active":"2024-03-30T13:01:41.987920+0000","last_became_peered":"2024-03-30T13:01:41.987920+0000","last_unstale":"2024-03-30T13:05:22.789864+0000","last_undegraded":"2024-03-30T13:05:22.789864+0000","last_fullsized":"2024-03-30T13:05:22.789864+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:05:13.431611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:05:13.431611+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:29.995673+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22789962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":200,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-30T13:05:22.790169+0000","last_change":"2024-03-30T13:05:07.441580+0000","last_active":"2024-03-30T13:05:22.790169+0000","last_peered":"2024-03-30T13:05:22.790169+0000","last_clean":"2024-03-30T13:05:22.790169+0000","last_became_active":"2024-03-30T13:05:07.440390+0000","last_became_peered":"2024-03-30T13:05:07.440390+0000","last_unstale":"2024-03-30T13:05:22.790169+0000","last_undegraded":"2024-03-30T13:05:22.790169+0000","last_fullsized":"2024-03-30T13:05:22.790169+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:04:27.400094+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:04:27.400094+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:51.289128+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":606,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-30T13:05:22.789283+0000","last_change":"2024-03-30T13:04:16.038106+0000","last_active":"2024-03-30T13:05:22.789283+0000","last_peered":"2024-03-30T13:05:22.789283+0000","last_clean":"2024-03-30T13:05:22.789283+0000","last_became_active":"2024-03-30T13:04:16.037485+0000","last_became_peered":"2024-03-30T13:04:16.037485+0000","last_unstale":"2024-03-30T13:05:22.789283+0000","last_undegraded":"2024-03-30T13:05:22.789283+0000","last_fullsized":"2024-03-30T13:05:22.789283+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:03:57.023686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:03:57.023686+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29378240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":672,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-30T13:05:22.789900+0000","last_change":"2024-03-30T13:04:40.339554+0000","last_active":"2024-03-30T13:05:22.789900+0000","last_peered":"2024-03-30T13:05:22.789900+0000","last_clean":"2024-03-30T13:05:22.789900+0000","last_became_active":"2024-03-30T13:03:25.980366+0000","last_became_peered":"2024-03-30T13:03:25.980366+0000","last_unstale":"2024-03-30T13:05:22.789900+0000","last_undegraded":"2024-03-30T13:05:22.789900+0000","last_fullsized":"2024-03-30T13:05:22.789900+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:40.339270+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:40.339270+0000","last_clean_scrub_stamp":"2024-03-30T13:04:40.339270+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:05.690409+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":45680832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":461,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-30T13:05:20.670819+0000","last_change":"2024-03-30T13:05:12.144999+0000","last_active":"2024-03-30T13:05:20.670819+0000","last_peered":"2024-03-30T13:05:20.670819+0000","last_clean":"2024-03-30T13:05:20.670819+0000","last_became_active":"2024-03-30T13:05:07.445793+0000","last_became_peered":"2024-03-30T13:05:07.445793+0000","last_unstale":"2024-03-30T13:05:20.670819+0000","last_undegraded":"2024-03-30T13:05:20.670819+0000","last_fullsized":"2024-03-30T13:05:20.670819+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:05:12.144734+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:05:12.144734+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:30.214751+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":625,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-03-30T13:05:24.628206+0000","last_change":"2024-03-30T13:05:07.446715+0000","last_active":"2024-03-30T13:05:24.628206+0000","last_peered":"2024-03-30T13:05:24.628206+0000","last_clean":"2024-03-30T13:05:24.628206+0000","last_became_active":"2024-03-30T13:05:07.445475+0000","last_became_peered":"2024-03-30T13:05:07.445475+0000","last_unstale":"2024-03-30T13:05:24.628206+0000","last_undegraded":"2024-03-30T13:05:24.628206+0000","last_fullsized":"2024-03-30T13:05:24.628206+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:05:00.265024+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:05:00.265024+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:17.234152+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":597,"reported_epoch":266,"state":"peering","last_fresh":"2024-03-30T13:05:22.618378+0000","last_change":"2024-03-30T13:05:24.629368+0000","last_active":"2024-03-30T13:05:24.629368+0000","last_peered":"2024-03-30T13:05:22.618378+0000","last_clean":"2024-03-30T13:05:22.618378+0000","last_became_active":"2024-03-30T13:04:16.049244+0000","last_became_peered":"2024-03-30T13:04:16.049244+0000","last_unstale":"2024-03-30T13:05:22.618378+0000","last_undegraded":"2024-03-30T13:05:22.618378+0000","last_fullsized":"2024-03-30T13:05:22.618378+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:04:53.372665+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:04:53.372665+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2147483647],"acting":[6,3,2147483647],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":34,"reported_epoch":267,"state":"peering","last_fresh":"2024-03-30T13:05:24.629580+0000","last_change":"2024-03-30T13:01:03.279540+0000","last_active":"2024-03-30T13:01:03.279540+0000","last_peered":"2024-03-30T12:55:47.287127+0000","last_clean":"2024-03-30T12:55:47.287127+0000","last_became_active":"2024-03-30T12:55:43.936322+0000","last_became_peered":"2024-03-30T12:55:43.936322+0000","last_unstale":"2024-03-30T13:05:24.629580+0000","last_undegraded":"2024-03-30T13:05:24.629580+0000","last_fullsized":"2024-03-30T13:05:24.629580+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_clean_scrub_stamp":"2024-03-30T13:04:21.312610+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,3],"acting":[2147483647,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":152,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-30T13:05:22.790401+0000","last_change":"2024-03-30T13:04:51.422166+0000","last_active":"2024-03-30T13:05:22.790401+0000","last_peered":"2024-03-30T13:05:22.790401+0000","last_clean":"2024-03-30T13:05:22.790401+0000","last_became_active":"2024-03-30T13:03:25.178234+0000","last_became_peered":"2024-03-30T13:03:25.178234+0000","last_unstale":"2024-03-30T13:05:22.790401+0000","last_undegraded":"2024-03-30T13:05:22.790401+0000","last_fullsized":"2024-03-30T13:05:22.790401+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:04:51.421910+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:04:51.421910+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:54.940198+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":34,"reported_epoch":267,"state":"peering","last_fresh":"2024-03-30T13:05:24.629762+0000","last_change":"2024-03-30T13:01:03.279945+0000","last_active":"2024-03-30T13:01:03.279945+0000","last_peered":"2024-03-30T12:55:47.343732+0000","last_clean":"2024-03-30T12:55:47.343732+0000","last_became_active":"2024-03-30T12:55:43.934661+0000","last_became_peered":"2024-03-30T12:55:43.934661+0000","last_unstale":"2024-03-30T13:05:24.629762+0000","last_undegraded":"2024-03-30T13:05:24.629762+0000","last_fullsized":"2024-03-30T13:05:24.629762+0000","mapping_epoch":267,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:05:22.298091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:05:22.298091+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,2,6],"acting":[2147483647,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":579,"reported_epoch":266,"state":"active+clean","last_fresh":"2024-03-30T13:05:22.789284+0000","last_change":"2024-03-30T13:04:55.119064+0000","last_active":"2024-03-30T13:05:22.789284+0000","last_peered":"2024-03-30T13:05:22.789284+0000","last_clean":"2024-03-30T13:05:22.789284+0000","last_became_active":"2024-03-30T13:01:41.986001+0000","last_became_peered":"2024-03-30T13:01:41.986001+0000","last_unstale":"2024-03-30T13:05:22.789284+0000","last_undegraded":"2024-03-30T13:05:22.789284+0000","last_fullsized":"2024-03-30T13:05:22.789284+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:55.118814+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:04:55.118814+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:17.644874+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10453822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":179059369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":227,"seq":974957576208,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49956,"kb_used_data":18252,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321884,"statfs":{"total":96636764160,"available":96585609216,"internally_reserved":0,"allocated":18690048,"data_stored":16908372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824005,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47264,"kb_used_data":14676,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94324576,"statfs":{"total":96636764160,"available":96588365824,"internally_reserved":0,"allocated":15028224,"data_stored":13260150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509562,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40660,"kb_used_data":9468,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94331180,"statfs":{"total":96636764160,"available":96595128320,"internally_reserved":0,"allocated":9695232,"data_stored":7913556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285647,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40000,"kb_used_data":8300,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94331840,"statfs":{"total":96636764160,"available":96595804160,"internally_reserved":0,"allocated":8499200,"data_stored":6724325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509562,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47824,"kb_used_data":16644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324016,"statfs":{"total":96636764160,"available":96587792384,"internally_reserved":0,"allocated":17043456,"data_stored":15261780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":206,"seq":884763263003,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46236,"kb_used_data":13188,"kb_used_omap":9,"kb_used_meta":33014,"kb_avail":94325604,"statfs":{"total":96636764160,"available":96589418496,"internally_reserved":0,"allocated":13504512,"data_stored":11722836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9338,"internal_metadata":33807238},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574970,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42660,"kb_used_data":15632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94329180,"statfs":{"total":96636764160,"available":96593080320,"internally_reserved":0,"allocated":16007168,"data_stored":14232293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:05:26.004 INFO:tasks.ceph.ceph_manager.ceph:PG 1.f is not active or peered 2024-03-30T13:05:26.005 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.f', 'version': "22'6", 'reported_seq': 571, 'reported_epoch': 266, 'state': 'peering', 'last_fresh': '2024-03-30T13:05:22.618379+0000', 'last_change': '2024-03-30T13:05:24.628882+0000', 'last_active': '2024-03-30T13:05:24.628882+0000', 'last_peered': '2024-03-30T13:05:22.618379+0000', 'last_clean': '2024-03-30T13:05:22.618379+0000', 'last_became_active': '2024-03-30T13:05:07.444573+0000', 'last_became_peered': '2024-03-30T13:05:07.444573+0000', 'last_unstale': '2024-03-30T13:05:22.618379+0000', 'last_undegraded': '2024-03-30T13:05:22.618379+0000', 'last_fullsized': '2024-03-30T13:05:22.618379+0000', 'mapping_epoch': 267, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 251, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "22'6", 'last_scrub_stamp': '2024-03-30T13:04:16.526295+0000', 'last_deep_scrub': "22'6", 'last_deep_scrub_stamp': '2024-03-30T13:04:16.526295+0000', 'last_clean_scrub_stamp': '2024-03-30T13:04:16.526295+0000', 'objects_scrubbed': 2, 'log_size': 6, 'log_dups_size': 0, 'ondisk_log_size': 6, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 7, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4588245, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 3, 'num_read_kb': 1, 'num_write': 8, 'num_write_kb': 1781, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4, 'num_bytes_recovered': 9176490, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2147483647, 4], 'acting': [6, 2147483647, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [4], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-30T13:05:26.005 INFO:tasks.ceph.ceph_manager.ceph:PG 1.1 is not active or peered 2024-03-30T13:05:26.005 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.1', 'version': "22'3", 'reported_seq': 22, 'reported_epoch': 267, 'state': 'peering', 'last_fresh': '2024-03-30T13:05:24.629507+0000', 'last_change': '2024-03-30T13:01:03.279745+0000', 'last_active': '2024-03-30T13:01:03.279745+0000', 'last_peered': '2024-03-30T12:55:47.164508+0000', 'last_clean': '2024-03-30T12:55:47.164508+0000', 'last_became_active': '2024-03-30T12:55:43.933770+0000', 'last_became_peered': '2024-03-30T12:55:43.933770+0000', 'last_unstale': '2024-03-30T13:05:24.629507+0000', 'last_undegraded': '2024-03-30T13:05:24.629507+0000', 'last_fullsized': '2024-03-30T13:05:24.629507+0000', 'mapping_epoch': 267, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 158, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "22'3", 'last_scrub_stamp': '2024-03-30T13:05:17.370216+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-30T12:55:42.900074+0000', 'last_clean_scrub_stamp': '2024-03-30T13:05:17.370216+0000', 'objects_scrubbed': 0, 'log_size': 3, 'log_dups_size': 0, 'ondisk_log_size': 3, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2235298, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 3, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 0, 'num_write': 4, 'num_write_kb': 914, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 2, 0], 'acting': [2147483647, 2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-30T13:05:26.005 INFO:tasks.ceph.ceph_manager.ceph:PG 1.7 is not active or peered 2024-03-30T13:05:26.005 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.7', 'version': "22'24", 'reported_seq': 597, 'reported_epoch': 266, 'state': 'peering', 'last_fresh': '2024-03-30T13:05:22.618378+0000', 'last_change': '2024-03-30T13:05:24.629368+0000', 'last_active': '2024-03-30T13:05:24.629368+0000', 'last_peered': '2024-03-30T13:05:22.618378+0000', 'last_clean': '2024-03-30T13:05:22.618378+0000', 'last_became_active': '2024-03-30T13:04:16.049244+0000', 'last_became_peered': '2024-03-30T13:04:16.049244+0000', 'last_unstale': '2024-03-30T13:05:22.618378+0000', 'last_undegraded': '2024-03-30T13:05:22.618378+0000', 'last_fullsized': '2024-03-30T13:05:22.618378+0000', 'mapping_epoch': 267, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 228, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "22'24", 'last_scrub_stamp': '2024-03-30T13:04:53.372665+0000', 'last_deep_scrub': "22'24", 'last_deep_scrub_stamp': '2024-03-30T13:03:28.655753+0000', 'last_clean_scrub_stamp': '2024-03-30T13:04:53.372665+0000', 'objects_scrubbed': 8, 'log_size': 24, 'log_dups_size': 0, 'ondisk_log_size': 24, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 7, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 19243751, 'num_objects': 8, 'num_object_clones': 0, 'num_object_copies': 24, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 8, 'num_whiteouts': 0, 'num_read': 15, 'num_read_kb': 7, 'num_write': 32, 'num_write_kb': 8385, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3, 2147483647], 'acting': [6, 3, 2147483647], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-30T13:05:26.005 INFO:tasks.ceph.ceph_manager.ceph:PG 1.8 is not active or peered 2024-03-30T13:05:26.005 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.8', 'version': "22'12", 'reported_seq': 34, 'reported_epoch': 267, 'state': 'peering', 'last_fresh': '2024-03-30T13:05:24.629580+0000', 'last_change': '2024-03-30T13:01:03.279540+0000', 'last_active': '2024-03-30T13:01:03.279540+0000', 'last_peered': '2024-03-30T12:55:47.287127+0000', 'last_clean': '2024-03-30T12:55:47.287127+0000', 'last_became_active': '2024-03-30T12:55:43.936322+0000', 'last_became_peered': '2024-03-30T12:55:43.936322+0000', 'last_unstale': '2024-03-30T13:05:24.629580+0000', 'last_undegraded': '2024-03-30T13:05:24.629580+0000', 'last_fullsized': '2024-03-30T13:05:24.629580+0000', 'mapping_epoch': 267, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 158, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "22'12", 'last_scrub_stamp': '2024-03-30T13:04:21.312610+0000', 'last_deep_scrub': "22'12", 'last_deep_scrub_stamp': '2024-03-30T13:04:21.312610+0000', 'last_clean_scrub_stamp': '2024-03-30T13:04:21.312610+0000', 'objects_scrubbed': 0, 'log_size': 12, 'log_dups_size': 0, 'ondisk_log_size': 12, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9159683, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 6, 'num_read_kb': 2, 'num_write': 16, 'num_write_kb': 3914, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 2, 3], 'acting': [2147483647, 2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-30T13:05:26.005 INFO:tasks.ceph.ceph_manager.ceph:PG 1.a is not active or peered 2024-03-30T13:05:26.005 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.a', 'version': "22'12", 'reported_seq': 34, 'reported_epoch': 267, 'state': 'peering', 'last_fresh': '2024-03-30T13:05:24.629762+0000', 'last_change': '2024-03-30T13:01:03.279945+0000', 'last_active': '2024-03-30T13:01:03.279945+0000', 'last_peered': '2024-03-30T12:55:47.343732+0000', 'last_clean': '2024-03-30T12:55:47.343732+0000', 'last_became_active': '2024-03-30T12:55:43.934661+0000', 'last_became_peered': '2024-03-30T12:55:43.934661+0000', 'last_unstale': '2024-03-30T13:05:24.629762+0000', 'last_undegraded': '2024-03-30T13:05:24.629762+0000', 'last_fullsized': '2024-03-30T13:05:24.629762+0000', 'mapping_epoch': 267, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 228, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "22'12", 'last_scrub_stamp': '2024-03-30T13:05:22.298091+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-30T12:55:42.900074+0000', 'last_clean_scrub_stamp': '2024-03-30T13:05:22.298091+0000', 'objects_scrubbed': 0, 'log_size': 12, 'log_dups_size': 0, 'ondisk_log_size': 12, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9075648, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 7, 'num_read_kb': 3, 'num_write': 16, 'num_write_kb': 4361, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 2, 6], 'acting': [2147483647, 2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-30T13:05:26.005 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:05:51.007 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:05:51.007 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:05:51.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:51.130+0000 7f09bfb7c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:51.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:05:51.142+0000 7f09bfb7c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:51.319 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:05:51.319 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:05:51.337 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":573,"stamp":"2024-03-30T13:05:49.957864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":252404094,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":334792,"kb_used_data":116408,"kb_used_omap":29,"kb_used_meta":218274,"kb_avail":660268088,"statfs":{"total":676457349120,"available":676114522112,"internally_reserved":0,"allocated":119201792,"data_stored":106536282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30616,"internal_metadata":223512680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002827"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":597,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-30T13:05:35.437669+0000","last_change":"2024-03-30T13:05:31.459925+0000","last_active":"2024-03-30T13:05:35.437669+0000","last_peered":"2024-03-30T13:05:35.437669+0000","last_clean":"2024-03-30T13:05:35.437669+0000","last_became_active":"2024-03-30T13:05:26.664389+0000","last_became_peered":"2024-03-30T13:05:26.664389+0000","last_unstale":"2024-03-30T13:05:35.437669+0000","last_undegraded":"2024-03-30T13:05:35.437669+0000","last_fullsized":"2024-03-30T13:05:35.437669+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:05:31.459608+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:05:31.459608+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:32.866650+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":13764735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":547,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-30T13:05:34.033002+0000","last_change":"2024-03-30T13:05:09.190113+0000","last_active":"2024-03-30T13:05:34.033002+0000","last_peered":"2024-03-30T13:05:34.033002+0000","last_clean":"2024-03-30T13:05:34.033002+0000","last_became_active":"2024-03-30T13:05:07.444962+0000","last_became_peered":"2024-03-30T13:05:07.444962+0000","last_unstale":"2024-03-30T13:05:34.033002+0000","last_undegraded":"2024-03-30T13:05:34.033002+0000","last_fullsized":"2024-03-30T13:05:34.033002+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:05:09.189831+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:05:09.189831+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:14.013241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":184,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-30T13:05:35.172340+0000","last_change":"2024-03-30T13:05:07.444027+0000","last_active":"2024-03-30T13:05:35.172340+0000","last_peered":"2024-03-30T13:05:35.172340+0000","last_clean":"2024-03-30T13:05:35.172340+0000","last_became_active":"2024-03-30T13:05:07.442521+0000","last_became_peered":"2024-03-30T13:05:07.442521+0000","last_unstale":"2024-03-30T13:05:35.172340+0000","last_undegraded":"2024-03-30T13:05:35.172340+0000","last_fullsized":"2024-03-30T13:05:35.172340+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:04:55.327587+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_clean_scrub_stamp":"2024-03-30T13:04:55.327587+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:10.849965+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":181,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-30T13:05:35.172191+0000","last_change":"2024-03-30T13:04:36.430290+0000","last_active":"2024-03-30T13:05:35.172191+0000","last_peered":"2024-03-30T13:05:35.172191+0000","last_clean":"2024-03-30T13:05:35.172191+0000","last_became_active":"2024-03-30T13:04:16.039346+0000","last_became_peered":"2024-03-30T13:04:16.039346+0000","last_unstale":"2024-03-30T13:05:35.172191+0000","last_undegraded":"2024-03-30T13:05:35.172191+0000","last_fullsized":"2024-03-30T13:05:35.172191+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:36.430045+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:04:36.430045+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:04.050679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":17,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-30T13:05:34.032742+0000","last_change":"2024-03-30T13:05:27.262936+0000","last_active":"2024-03-30T13:05:34.032742+0000","last_peered":"2024-03-30T13:05:34.032742+0000","last_clean":"2024-03-30T13:05:34.032742+0000","last_became_active":"2024-03-30T13:05:26.663586+0000","last_became_peered":"2024-03-30T13:05:26.663586+0000","last_unstale":"2024-03-30T13:05:34.032742+0000","last_undegraded":"2024-03-30T13:05:34.032742+0000","last_fullsized":"2024-03-30T13:05:34.032742+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:05:17.370216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:05:17.370216+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:45.530655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":627,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-30T13:05:35.298574+0000","last_change":"2024-03-30T13:05:27.740232+0000","last_active":"2024-03-30T13:05:35.298574+0000","last_peered":"2024-03-30T13:05:35.298574+0000","last_clean":"2024-03-30T13:05:35.298574+0000","last_became_active":"2024-03-30T13:05:26.671956+0000","last_became_peered":"2024-03-30T13:05:26.671956+0000","last_unstale":"2024-03-30T13:05:35.298574+0000","last_undegraded":"2024-03-30T13:05:35.298574+0000","last_fullsized":"2024-03-30T13:05:35.298574+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:05:13.431611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:05:13.431611+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:21.065109+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":209,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-30T13:05:35.172403+0000","last_change":"2024-03-30T13:05:07.441580+0000","last_active":"2024-03-30T13:05:35.172403+0000","last_peered":"2024-03-30T13:05:35.172403+0000","last_clean":"2024-03-30T13:05:35.172403+0000","last_became_active":"2024-03-30T13:05:07.440390+0000","last_became_peered":"2024-03-30T13:05:07.440390+0000","last_unstale":"2024-03-30T13:05:35.172403+0000","last_undegraded":"2024-03-30T13:05:35.172403+0000","last_fullsized":"2024-03-30T13:05:35.172403+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:04:27.400094+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:04:27.400094+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:51.289128+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":635,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-30T13:05:34.688431+0000","last_change":"2024-03-30T13:05:26.740983+0000","last_active":"2024-03-30T13:05:34.688431+0000","last_peered":"2024-03-30T13:05:34.688431+0000","last_clean":"2024-03-30T13:05:34.688431+0000","last_became_active":"2024-03-30T13:05:26.671576+0000","last_became_peered":"2024-03-30T13:05:26.671576+0000","last_unstale":"2024-03-30T13:05:34.688431+0000","last_undegraded":"2024-03-30T13:05:34.688431+0000","last_fullsized":"2024-03-30T13:05:34.688431+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:05:24.040489+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:47.369913+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":36722800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":699,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-30T13:05:45.457333+0000","last_change":"2024-03-30T13:05:45.457410+0000","last_active":"2024-03-30T13:05:45.457333+0000","last_peered":"2024-03-30T13:05:45.457333+0000","last_clean":"2024-03-30T13:05:45.457333+0000","last_became_active":"2024-03-30T13:05:26.674745+0000","last_became_peered":"2024-03-30T13:05:26.674745+0000","last_unstale":"2024-03-30T13:05:45.457333+0000","last_undegraded":"2024-03-30T13:05:45.457333+0000","last_fullsized":"2024-03-30T13:05:45.457333+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:05:45.457140+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:40.339270+0000","last_clean_scrub_stamp":"2024-03-30T13:05:45.457140+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:09.231195+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":50756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":473,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-30T13:05:34.032795+0000","last_change":"2024-03-30T13:05:12.144999+0000","last_active":"2024-03-30T13:05:34.032795+0000","last_peered":"2024-03-30T13:05:34.032795+0000","last_clean":"2024-03-30T13:05:34.032795+0000","last_became_active":"2024-03-30T13:05:07.445793+0000","last_became_peered":"2024-03-30T13:05:07.445793+0000","last_unstale":"2024-03-30T13:05:34.032795+0000","last_undegraded":"2024-03-30T13:05:34.032795+0000","last_fullsized":"2024-03-30T13:05:34.032795+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:05:12.144734+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:05:12.144734+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:30.214751+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":631,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-30T13:05:35.438044+0000","last_change":"2024-03-30T13:05:07.446715+0000","last_active":"2024-03-30T13:05:35.438044+0000","last_peered":"2024-03-30T13:05:35.438044+0000","last_clean":"2024-03-30T13:05:35.438044+0000","last_became_active":"2024-03-30T13:05:07.445475+0000","last_became_peered":"2024-03-30T13:05:07.445475+0000","last_unstale":"2024-03-30T13:05:35.438044+0000","last_undegraded":"2024-03-30T13:05:35.438044+0000","last_fullsized":"2024-03-30T13:05:35.438044+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:05:00.265024+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:05:00.265024+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:17.234152+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":627,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-30T13:05:35.438136+0000","last_change":"2024-03-30T13:05:28.475431+0000","last_active":"2024-03-30T13:05:35.438136+0000","last_peered":"2024-03-30T13:05:35.438136+0000","last_clean":"2024-03-30T13:05:35.438136+0000","last_became_active":"2024-03-30T13:05:26.669714+0000","last_became_peered":"2024-03-30T13:05:26.669714+0000","last_unstale":"2024-03-30T13:05:35.438136+0000","last_undegraded":"2024-03-30T13:05:35.438136+0000","last_fullsized":"2024-03-30T13:05:35.438136+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:04:53.372665+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:04:53.372665+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:00.512149+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":31,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-30T13:05:35.437585+0000","last_change":"2024-03-30T13:05:28.485094+0000","last_active":"2024-03-30T13:05:35.437585+0000","last_peered":"2024-03-30T13:05:35.437585+0000","last_clean":"2024-03-30T13:05:35.437585+0000","last_became_active":"2024-03-30T13:05:26.668304+0000","last_became_peered":"2024-03-30T13:05:26.668304+0000","last_unstale":"2024-03-30T13:05:35.437585+0000","last_undegraded":"2024-03-30T13:05:35.437585+0000","last_fullsized":"2024-03-30T13:05:35.437585+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:05:28.484860+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_clean_scrub_stamp":"2024-03-30T13:05:28.484860+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:57.917739+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":161,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-30T13:05:35.172191+0000","last_change":"2024-03-30T13:04:51.422166+0000","last_active":"2024-03-30T13:05:35.172191+0000","last_peered":"2024-03-30T13:05:35.172191+0000","last_clean":"2024-03-30T13:05:35.172191+0000","last_became_active":"2024-03-30T13:03:25.178234+0000","last_became_peered":"2024-03-30T13:03:25.178234+0000","last_unstale":"2024-03-30T13:05:35.172191+0000","last_undegraded":"2024-03-30T13:05:35.172191+0000","last_fullsized":"2024-03-30T13:05:35.172191+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:04:51.421910+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:04:51.421910+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:05:54.940198+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":23,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-30T13:05:34.032729+0000","last_change":"2024-03-30T13:05:27.379613+0000","last_active":"2024-03-30T13:05:34.032729+0000","last_peered":"2024-03-30T13:05:34.032729+0000","last_clean":"2024-03-30T13:05:34.032729+0000","last_became_active":"2024-03-30T13:05:26.663719+0000","last_became_peered":"2024-03-30T13:05:26.663719+0000","last_unstale":"2024-03-30T13:05:34.032729+0000","last_undegraded":"2024-03-30T13:05:34.032729+0000","last_fullsized":"2024-03-30T13:05:34.032729+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:05:22.298091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:05:22.298091+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:49.334835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":599,"reported_epoch":270,"state":"active+clean","last_fresh":"2024-03-30T13:05:34.688487+0000","last_change":"2024-03-30T13:05:27.364417+0000","last_active":"2024-03-30T13:05:34.688487+0000","last_peered":"2024-03-30T13:05:34.688487+0000","last_clean":"2024-03-30T13:05:34.688487+0000","last_became_active":"2024-03-30T13:05:26.670891+0000","last_became_peered":"2024-03-30T13:05:26.670891+0000","last_unstale":"2024-03-30T13:05:34.688487+0000","last_undegraded":"2024-03-30T13:05:34.688487+0000","last_fullsized":"2024-03-30T13:05:34.688487+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:55.118814+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:04:55.118814+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:16.781359+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15680733,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":252404094,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":116637696,"data_stored":116637696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":227,"seq":974957576212,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55556,"kb_used_data":23864,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94316284,"statfs":{"total":96636764160,"available":96579874816,"internally_reserved":0,"allocated":24436736,"data_stored":22628343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824010,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51832,"kb_used_data":19244,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94320008,"statfs":{"total":96636764160,"available":96583688192,"internally_reserved":0,"allocated":19705856,"data_stored":17897463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509567,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43336,"kb_used_data":12156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328504,"statfs":{"total":96636764160,"available":96592388096,"internally_reserved":0,"allocated":12447744,"data_stored":10632592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285652,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45264,"kb_used_data":13572,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94326576,"statfs":{"total":96636764160,"available":96590413824,"internally_reserved":0,"allocated":13897728,"data_stored":12089335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509567,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47872,"kb_used_data":16692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323968,"statfs":{"total":96636764160,"available":96587743232,"internally_reserved":0,"allocated":17092608,"data_stored":15284215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":206,"seq":884763263008,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46272,"kb_used_data":13236,"kb_used_omap":9,"kb_used_meta":33014,"kb_avail":94325568,"statfs":{"total":96636764160,"available":96589381632,"internally_reserved":0,"allocated":13553664,"data_stored":11745271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9338,"internal_metadata":33807238},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574975,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44660,"kb_used_data":17644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94327180,"statfs":{"total":96636764160,"available":96591032320,"internally_reserved":0,"allocated":18067456,"data_stored":16259063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10629120,"data_stored":10629120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9166848,"data_stored":9166848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":14798848,"data_stored":14798848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16437248,"data_stored":16437248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":21168128,"data_stored":21168128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:05:51.338 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:05:51.338 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:05:51.338 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-03-30T13:05:51.338 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-30T13:05:51.339 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-30T13:05:51.341 INFO:tasks.ceph.osd.5:Started 2024-03-30T13:05:51.342 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-30T13:05:51.411 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:05:51.409+0000 7f5b615447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:51.419 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:05:51.417+0000 7f5b615447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:51.419 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:05:51.417+0000 7f5b615447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:05:51.463 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T13:05:51.463 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T13:05:51.463 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T13:05:51.463 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T13:05:51.463 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T13:05:51.463 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T13:05:51.463 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-03-30T13:05:51.463 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-03-30T13:05:51.464 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-03-30T13:05:51.464 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-03-30T13:05:51.464 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-03-30T13:05:51.464 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T13:05:51.466 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:05:51.466 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-03-30T13:05:51.974 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:05:51.973+0000 7f5b615447c0 -1 Falling back to public interface 2024-03-30T13:05:55.731 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:05:55.729+0000 7f5b615447c0 -1 osd.5 266 log_to_monitors true 2024-03-30T13:05:56.467 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-30T13:05:56.592 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T13:05:56.592 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T13:05:56.592 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T13:05:56.592 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T13:05:56.593 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T13:05:56.593 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T13:05:56.593 INFO:teuthology.orchestra.run.smithi082.stderr:bluefs debug_inject_read_zeros 2024-03-30T13:05:56.593 INFO:teuthology.orchestra.run.smithi082.stderr:bluefs files list 2024-03-30T13:05:56.593 INFO:teuthology.orchestra.run.smithi082.stderr:bluefs stats 2024-03-30T13:05:56.593 INFO:teuthology.orchestra.run.smithi082.stderr:bluestore allocator dump block 2024-03-30T13:05:56.593 INFO:teuthology.orchestra.run.smithi082.stderr:bluestore allocator fragmentation block 2024-03-30T13:05:56.593 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T13:05:56.595 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:05:56.595 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-03-30T13:05:57.162 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:05:57.161+0000 7f5b55311640 -1 osd.5 266 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:06:01.596 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-30T13:06:01.727 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T13:06:01.727 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T13:06:01.727 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T13:06:01.727 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T13:06:01.741 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:06:01.741 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-03-30T13:06:01.741 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 5 2024-03-30T13:06:01.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:01.870+0000 7f4593d82640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:01.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:01.882+0000 7f4593d82640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:02.064 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:06:02.066+0000 7fbb38f08640 -1 mon.a@0(leader).osd e275 definitely_dead 0 2024-03-30T13:06:02.174 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.5. 2024-03-30T13:06:02.201 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-03-30T13:06:02.202 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:06:02.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:02.318+0000 7f1e0e39a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:02.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:02.334+0000 7f1e0e39a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:03.185 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:06:03.217 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:06:03.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:03.342+0000 7fa432ece640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:03.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:03.362+0000 7fa432ece640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:04.206 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:06:04.231 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:06:04.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:04.350+0000 7fa04b05e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:04.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:04.370+0000 7fa04b05e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:05.211 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:06:05.234 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:06:05.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:05.354+0000 7f237469c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:05.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:05.370+0000 7f237469c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:06.215 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:06:06.242 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:06:06.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:06.362+0000 7f65561c0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:06.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:06.378+0000 7f65561c0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:07.233 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:06:07.259 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:06:07.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:07.382+0000 7f36b5c1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:07.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:07.398+0000 7f36b5c1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:08.244 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:06:08.269 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:06:08.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:08.386+0000 7f8e067f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:08.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:08.402+0000 7f8e067f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:09.250 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:06:09.277 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:06:09.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:09.398+0000 7fe397c6c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:09.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:09.414+0000 7fe397c6c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:10.267 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:06:10.295 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:06:10.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:10.418+0000 7fa8ad3d2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:10.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:10.434+0000 7fa8ad3d2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:11.269 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:06:11.296 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:06:11.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:11.418+0000 7f5c8c152640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:11.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:11.434+0000 7f5c8c152640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:12.272 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:06:12.303 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:06:12.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:12.426+0000 7f1f62990640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:12.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:12.442+0000 7f1f62990640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:13.282 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:06:13.310 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:06:13.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:13.434+0000 7fbde427d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:13.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:13.450+0000 7fbde427d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:14.291 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:06:14.316 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:06:14.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:14.434+0000 7f03436a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:14.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:14.450+0000 7f03436a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:15.300 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:06:15.328 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:06:15.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:15.450+0000 7fd05e6cb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:15.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:15.466+0000 7fd05e6cb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:16.317 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:06:16.346 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:06:16.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:16.470+0000 7fa6ae565640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:16.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:16.486+0000 7fa6ae565640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:17.321 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:06:17.347 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:06:17.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:17.466+0000 7f43bd504640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:17.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:17.682+0000 7f43b77fe640 0 --2- 172.21.15.67:0/1538592951 >> v2:172.21.15.67:3300/0 conn(0x7f43b813c170 0x7f43b813c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-03-30T13:06:17.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:17.686+0000 7f43bd504640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:18.342 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:06:18.367 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:06:18.368 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:06:18.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:18.494+0000 7fda20f80640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:18.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:18.506+0000 7fda20f80640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:18.681 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:06:18.682 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:06:18.700 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":608,"stamp":"2024-03-30T13:06:18.340912+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":305278190,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":376076,"kb_used_data":121352,"kb_used_omap":35,"kb_used_meta":254556,"kb_avail":754598644,"statfs":{"total":773094113280,"available":772709011456,"internally_reserved":0,"allocated":124264448,"data_stored":108812341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36669,"internal_metadata":260665539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.045995"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":642,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-03-30T13:06:14.249041+0000","last_change":"2024-03-30T13:06:03.258135+0000","last_active":"2024-03-30T13:06:14.249041+0000","last_peered":"2024-03-30T13:06:14.249041+0000","last_clean":"2024-03-30T13:06:14.249041+0000","last_became_active":"2024-03-30T13:06:03.198084+0000","last_became_peered":"2024-03-30T13:06:03.198084+0000","last_unstale":"2024-03-30T13:06:14.249041+0000","last_undegraded":"2024-03-30T13:06:14.249041+0000","last_fullsized":"2024-03-30T13:06:14.249041+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:05:31.459608+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:05:31.459608+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:33.232176+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18352980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":595,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-03-30T13:06:16.317673+0000","last_change":"2024-03-30T13:06:14.050179+0000","last_active":"2024-03-30T13:06:16.317673+0000","last_peered":"2024-03-30T13:06:16.317673+0000","last_clean":"2024-03-30T13:06:16.317673+0000","last_became_active":"2024-03-30T13:05:07.444962+0000","last_became_peered":"2024-03-30T13:05:07.444962+0000","last_unstale":"2024-03-30T13:06:16.317673+0000","last_undegraded":"2024-03-30T13:06:16.317673+0000","last_fullsized":"2024-03-30T13:06:16.317673+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:06:14.049898+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:06:14.049898+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:39.577970+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":225,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-03-30T13:06:14.045599+0000","last_change":"2024-03-30T13:06:11.665355+0000","last_active":"2024-03-30T13:06:14.045599+0000","last_peered":"2024-03-30T13:06:14.045599+0000","last_clean":"2024-03-30T13:06:14.045599+0000","last_became_active":"2024-03-30T13:05:07.442521+0000","last_became_peered":"2024-03-30T13:05:07.442521+0000","last_unstale":"2024-03-30T13:06:14.045599+0000","last_undegraded":"2024-03-30T13:06:14.045599+0000","last_fullsized":"2024-03-30T13:06:14.045599+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:06:11.665099+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_clean_scrub_stamp":"2024-03-30T13:06:11.665099+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:34.778370+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":222,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-03-30T13:06:14.045496+0000","last_change":"2024-03-30T13:06:06.768447+0000","last_active":"2024-03-30T13:06:14.045496+0000","last_peered":"2024-03-30T13:06:14.045496+0000","last_clean":"2024-03-30T13:06:14.045496+0000","last_became_active":"2024-03-30T13:04:16.039346+0000","last_became_peered":"2024-03-30T13:04:16.039346+0000","last_unstale":"2024-03-30T13:06:14.045496+0000","last_undegraded":"2024-03-30T13:06:14.045496+0000","last_fullsized":"2024-03-30T13:06:14.045496+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:06:06.768191+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:06:06.768191+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:34.667227+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":43,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.795948+0000","last_change":"2024-03-30T13:06:03.311279+0000","last_active":"2024-03-30T13:06:17.795948+0000","last_peered":"2024-03-30T13:06:17.795948+0000","last_clean":"2024-03-30T13:06:17.795948+0000","last_became_active":"2024-03-30T13:06:03.203196+0000","last_became_peered":"2024-03-30T13:06:03.203196+0000","last_unstale":"2024-03-30T13:06:17.795948+0000","last_undegraded":"2024-03-30T13:06:17.795948+0000","last_fullsized":"2024-03-30T13:06:17.795948+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:05:17.370216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:05:17.370216+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:28.258191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":686,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.777471+0000","last_change":"2024-03-30T13:06:03.664841+0000","last_active":"2024-03-30T13:06:17.777471+0000","last_peered":"2024-03-30T13:06:17.777471+0000","last_clean":"2024-03-30T13:06:17.777471+0000","last_became_active":"2024-03-30T13:06:03.202467+0000","last_became_peered":"2024-03-30T13:06:03.202467+0000","last_unstale":"2024-03-30T13:06:17.777471+0000","last_undegraded":"2024-03-30T13:06:17.777471+0000","last_fullsized":"2024-03-30T13:06:17.777471+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:05:13.431611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:05:13.431611+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:36.582588+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":45579924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":250,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-03-30T13:06:14.045585+0000","last_change":"2024-03-30T13:05:54.746839+0000","last_active":"2024-03-30T13:06:14.045585+0000","last_peered":"2024-03-30T13:06:14.045585+0000","last_clean":"2024-03-30T13:06:14.045585+0000","last_became_active":"2024-03-30T13:05:07.440390+0000","last_became_peered":"2024-03-30T13:05:07.440390+0000","last_unstale":"2024-03-30T13:06:14.045585+0000","last_undegraded":"2024-03-30T13:06:14.045585+0000","last_fullsized":"2024-03-30T13:06:14.045585+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:05:54.746578+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:05:54.746578+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:11.361732+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":690,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.328432+0000","last_change":"2024-03-30T13:06:03.273364+0000","last_active":"2024-03-30T13:06:17.328432+0000","last_peered":"2024-03-30T13:06:17.328432+0000","last_clean":"2024-03-30T13:06:17.328432+0000","last_became_active":"2024-03-30T13:06:03.206177+0000","last_became_peered":"2024-03-30T13:06:03.206177+0000","last_unstale":"2024-03-30T13:06:17.328432+0000","last_undegraded":"2024-03-30T13:06:17.328432+0000","last_fullsized":"2024-03-30T13:06:17.328432+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:05:24.040489+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:25.404654+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":753,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.777475+0000","last_change":"2024-03-30T13:06:03.665679+0000","last_active":"2024-03-30T13:06:17.777475+0000","last_peered":"2024-03-30T13:06:17.777475+0000","last_clean":"2024-03-30T13:06:17.777475+0000","last_became_active":"2024-03-30T13:06:03.219523+0000","last_became_peered":"2024-03-30T13:06:03.219523+0000","last_unstale":"2024-03-30T13:06:17.777475+0000","last_undegraded":"2024-03-30T13:06:17.777475+0000","last_fullsized":"2024-03-30T13:06:17.777475+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:05:45.457140+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:40.339270+0000","last_clean_scrub_stamp":"2024-03-30T13:05:45.457140+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:08.613970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":55832128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":513,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-03-30T13:06:16.317638+0000","last_change":"2024-03-30T13:05:12.144999+0000","last_active":"2024-03-30T13:06:16.317638+0000","last_peered":"2024-03-30T13:06:16.317638+0000","last_clean":"2024-03-30T13:06:16.317638+0000","last_became_active":"2024-03-30T13:05:07.445793+0000","last_became_peered":"2024-03-30T13:05:07.445793+0000","last_unstale":"2024-03-30T13:06:16.317638+0000","last_undegraded":"2024-03-30T13:06:16.317638+0000","last_fullsized":"2024-03-30T13:06:16.317638+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:05:12.144734+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:05:12.144734+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:30.214751+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":664,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-03-30T13:06:14.248918+0000","last_change":"2024-03-30T13:05:07.446715+0000","last_active":"2024-03-30T13:06:14.248918+0000","last_peered":"2024-03-30T13:06:14.248918+0000","last_clean":"2024-03-30T13:06:14.248918+0000","last_became_active":"2024-03-30T13:05:07.445475+0000","last_became_peered":"2024-03-30T13:05:07.445475+0000","last_unstale":"2024-03-30T13:06:14.248918+0000","last_undegraded":"2024-03-30T13:06:14.248918+0000","last_fullsized":"2024-03-30T13:06:14.248918+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:05:00.265024+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:05:00.265024+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:17.234152+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":692,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-03-30T13:06:14.249008+0000","last_change":"2024-03-30T13:06:05.397483+0000","last_active":"2024-03-30T13:06:14.249008+0000","last_peered":"2024-03-30T13:06:14.249008+0000","last_clean":"2024-03-30T13:06:14.249008+0000","last_became_active":"2024-03-30T13:06:03.206037+0000","last_became_peered":"2024-03-30T13:06:03.206037+0000","last_unstale":"2024-03-30T13:06:14.249008+0000","last_undegraded":"2024-03-30T13:06:14.249008+0000","last_fullsized":"2024-03-30T13:06:14.249008+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:06:05.397104+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:06:05.397104+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:28.298853+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":49,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.795606+0000","last_change":"2024-03-30T13:06:03.317478+0000","last_active":"2024-03-30T13:06:17.795606+0000","last_peered":"2024-03-30T13:06:17.795606+0000","last_clean":"2024-03-30T13:06:17.795606+0000","last_became_active":"2024-03-30T13:06:03.194120+0000","last_became_peered":"2024-03-30T13:06:03.194120+0000","last_unstale":"2024-03-30T13:06:17.795606+0000","last_undegraded":"2024-03-30T13:06:17.795606+0000","last_fullsized":"2024-03-30T13:06:17.795606+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:05:28.484860+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_clean_scrub_stamp":"2024-03-30T13:05:28.484860+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:43.514196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":202,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-03-30T13:06:14.045587+0000","last_change":"2024-03-30T13:05:55.776927+0000","last_active":"2024-03-30T13:06:14.045587+0000","last_peered":"2024-03-30T13:06:14.045587+0000","last_clean":"2024-03-30T13:06:14.045587+0000","last_became_active":"2024-03-30T13:03:25.178234+0000","last_became_peered":"2024-03-30T13:03:25.178234+0000","last_unstale":"2024-03-30T13:06:14.045587+0000","last_undegraded":"2024-03-30T13:06:14.045587+0000","last_fullsized":"2024-03-30T13:06:14.045587+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:05:55.776693+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:05:55.776693+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:59.319019+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":49,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.795663+0000","last_change":"2024-03-30T13:06:03.294781+0000","last_active":"2024-03-30T13:06:17.795663+0000","last_peered":"2024-03-30T13:06:17.795663+0000","last_clean":"2024-03-30T13:06:17.795663+0000","last_became_active":"2024-03-30T13:06:03.198523+0000","last_became_peered":"2024-03-30T13:06:03.198523+0000","last_unstale":"2024-03-30T13:06:17.795663+0000","last_undegraded":"2024-03-30T13:06:17.795663+0000","last_fullsized":"2024-03-30T13:06:17.795663+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:05:22.298091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:05:22.298091+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:50.590330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":652,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.328560+0000","last_change":"2024-03-30T13:06:03.289927+0000","last_active":"2024-03-30T13:06:17.328560+0000","last_peered":"2024-03-30T13:06:17.328560+0000","last_clean":"2024-03-30T13:06:17.328560+0000","last_became_active":"2024-03-30T13:06:03.206122+0000","last_became_peered":"2024-03-30T13:06:03.206122+0000","last_unstale":"2024-03-30T13:06:17.328560+0000","last_undegraded":"2024-03-30T13:06:17.328560+0000","last_fullsized":"2024-03-30T13:06:17.328560+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:55.118814+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:04:55.118814+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:23.842135+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":20907644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":305278190,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":227,"seq":974957576218,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50196,"kb_used_data":18504,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321644,"statfs":{"total":96636764160,"available":96585363456,"internally_reserved":0,"allocated":18948096,"data_stored":17025934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":274,"seq":1176821039110,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55432,"kb_used_data":23284,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94316408,"statfs":{"total":96636764160,"available":96580001792,"internally_reserved":0,"allocated":23842816,"data_stored":21900251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6053,"internal_metadata":32893019},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824016,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47568,"kb_used_data":14976,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94324272,"statfs":{"total":96636764160,"available":96588054528,"internally_reserved":0,"allocated":15335424,"data_stored":13399660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509572,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40916,"kb_used_data":9732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330924,"statfs":{"total":96636764160,"available":96594866176,"internally_reserved":0,"allocated":9965568,"data_stored":8036605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285658,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40292,"kb_used_data":8600,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94331548,"statfs":{"total":96636764160,"available":96595505152,"internally_reserved":0,"allocated":8806400,"data_stored":6863835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509572,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48084,"kb_used_data":16896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323756,"statfs":{"total":96636764160,"available":96587526144,"internally_reserved":0,"allocated":17301504,"data_stored":15379342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":206,"seq":884763263013,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46468,"kb_used_data":13428,"kb_used_omap":9,"kb_used_meta":33014,"kb_avail":94325372,"statfs":{"total":96636764160,"available":96589180928,"internally_reserved":0,"allocated":13750272,"data_stored":11834911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9338,"internal_metadata":33807238},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574981,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47120,"kb_used_data":15932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324720,"statfs":{"total":96636764160,"available":96588513280,"internally_reserved":0,"allocated":16314368,"data_stored":14371803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:06:18.702 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:06:18.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:18.822+0000 7efd66baa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:18.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:18.838+0000 7efd66baa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:19.007 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:06:19.008 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:06:19.025 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":608,"stamp":"2024-03-30T13:06:18.340912+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":305278190,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":376076,"kb_used_data":121352,"kb_used_omap":35,"kb_used_meta":254556,"kb_avail":754598644,"statfs":{"total":773094113280,"available":772709011456,"internally_reserved":0,"allocated":124264448,"data_stored":108812341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36669,"internal_metadata":260665539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.045995"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":642,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-03-30T13:06:14.249041+0000","last_change":"2024-03-30T13:06:03.258135+0000","last_active":"2024-03-30T13:06:14.249041+0000","last_peered":"2024-03-30T13:06:14.249041+0000","last_clean":"2024-03-30T13:06:14.249041+0000","last_became_active":"2024-03-30T13:06:03.198084+0000","last_became_peered":"2024-03-30T13:06:03.198084+0000","last_unstale":"2024-03-30T13:06:14.249041+0000","last_undegraded":"2024-03-30T13:06:14.249041+0000","last_fullsized":"2024-03-30T13:06:14.249041+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:05:31.459608+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:05:31.459608+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:33.232176+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18352980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":595,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-03-30T13:06:16.317673+0000","last_change":"2024-03-30T13:06:14.050179+0000","last_active":"2024-03-30T13:06:16.317673+0000","last_peered":"2024-03-30T13:06:16.317673+0000","last_clean":"2024-03-30T13:06:16.317673+0000","last_became_active":"2024-03-30T13:05:07.444962+0000","last_became_peered":"2024-03-30T13:05:07.444962+0000","last_unstale":"2024-03-30T13:06:16.317673+0000","last_undegraded":"2024-03-30T13:06:16.317673+0000","last_fullsized":"2024-03-30T13:06:16.317673+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:06:14.049898+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:06:14.049898+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:39.577970+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":225,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-03-30T13:06:14.045599+0000","last_change":"2024-03-30T13:06:11.665355+0000","last_active":"2024-03-30T13:06:14.045599+0000","last_peered":"2024-03-30T13:06:14.045599+0000","last_clean":"2024-03-30T13:06:14.045599+0000","last_became_active":"2024-03-30T13:05:07.442521+0000","last_became_peered":"2024-03-30T13:05:07.442521+0000","last_unstale":"2024-03-30T13:06:14.045599+0000","last_undegraded":"2024-03-30T13:06:14.045599+0000","last_fullsized":"2024-03-30T13:06:14.045599+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:06:11.665099+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_clean_scrub_stamp":"2024-03-30T13:06:11.665099+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:34.778370+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":222,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-03-30T13:06:14.045496+0000","last_change":"2024-03-30T13:06:06.768447+0000","last_active":"2024-03-30T13:06:14.045496+0000","last_peered":"2024-03-30T13:06:14.045496+0000","last_clean":"2024-03-30T13:06:14.045496+0000","last_became_active":"2024-03-30T13:04:16.039346+0000","last_became_peered":"2024-03-30T13:04:16.039346+0000","last_unstale":"2024-03-30T13:06:14.045496+0000","last_undegraded":"2024-03-30T13:06:14.045496+0000","last_fullsized":"2024-03-30T13:06:14.045496+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:06:06.768191+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:06:06.768191+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:34.667227+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":43,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.795948+0000","last_change":"2024-03-30T13:06:03.311279+0000","last_active":"2024-03-30T13:06:17.795948+0000","last_peered":"2024-03-30T13:06:17.795948+0000","last_clean":"2024-03-30T13:06:17.795948+0000","last_became_active":"2024-03-30T13:06:03.203196+0000","last_became_peered":"2024-03-30T13:06:03.203196+0000","last_unstale":"2024-03-30T13:06:17.795948+0000","last_undegraded":"2024-03-30T13:06:17.795948+0000","last_fullsized":"2024-03-30T13:06:17.795948+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:05:17.370216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:05:17.370216+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:28.258191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":686,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.777471+0000","last_change":"2024-03-30T13:06:03.664841+0000","last_active":"2024-03-30T13:06:17.777471+0000","last_peered":"2024-03-30T13:06:17.777471+0000","last_clean":"2024-03-30T13:06:17.777471+0000","last_became_active":"2024-03-30T13:06:03.202467+0000","last_became_peered":"2024-03-30T13:06:03.202467+0000","last_unstale":"2024-03-30T13:06:17.777471+0000","last_undegraded":"2024-03-30T13:06:17.777471+0000","last_fullsized":"2024-03-30T13:06:17.777471+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:05:13.431611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:05:13.431611+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:36.582588+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":45579924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":250,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-03-30T13:06:14.045585+0000","last_change":"2024-03-30T13:05:54.746839+0000","last_active":"2024-03-30T13:06:14.045585+0000","last_peered":"2024-03-30T13:06:14.045585+0000","last_clean":"2024-03-30T13:06:14.045585+0000","last_became_active":"2024-03-30T13:05:07.440390+0000","last_became_peered":"2024-03-30T13:05:07.440390+0000","last_unstale":"2024-03-30T13:06:14.045585+0000","last_undegraded":"2024-03-30T13:06:14.045585+0000","last_fullsized":"2024-03-30T13:06:14.045585+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:05:54.746578+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:05:54.746578+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:11.361732+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":690,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.328432+0000","last_change":"2024-03-30T13:06:03.273364+0000","last_active":"2024-03-30T13:06:17.328432+0000","last_peered":"2024-03-30T13:06:17.328432+0000","last_clean":"2024-03-30T13:06:17.328432+0000","last_became_active":"2024-03-30T13:06:03.206177+0000","last_became_peered":"2024-03-30T13:06:03.206177+0000","last_unstale":"2024-03-30T13:06:17.328432+0000","last_undegraded":"2024-03-30T13:06:17.328432+0000","last_fullsized":"2024-03-30T13:06:17.328432+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:05:24.040489+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:25.404654+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":753,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.777475+0000","last_change":"2024-03-30T13:06:03.665679+0000","last_active":"2024-03-30T13:06:17.777475+0000","last_peered":"2024-03-30T13:06:17.777475+0000","last_clean":"2024-03-30T13:06:17.777475+0000","last_became_active":"2024-03-30T13:06:03.219523+0000","last_became_peered":"2024-03-30T13:06:03.219523+0000","last_unstale":"2024-03-30T13:06:17.777475+0000","last_undegraded":"2024-03-30T13:06:17.777475+0000","last_fullsized":"2024-03-30T13:06:17.777475+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:05:45.457140+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:40.339270+0000","last_clean_scrub_stamp":"2024-03-30T13:05:45.457140+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:08.613970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":55832128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":513,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-03-30T13:06:16.317638+0000","last_change":"2024-03-30T13:05:12.144999+0000","last_active":"2024-03-30T13:06:16.317638+0000","last_peered":"2024-03-30T13:06:16.317638+0000","last_clean":"2024-03-30T13:06:16.317638+0000","last_became_active":"2024-03-30T13:05:07.445793+0000","last_became_peered":"2024-03-30T13:05:07.445793+0000","last_unstale":"2024-03-30T13:06:16.317638+0000","last_undegraded":"2024-03-30T13:06:16.317638+0000","last_fullsized":"2024-03-30T13:06:16.317638+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:05:12.144734+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:05:12.144734+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:30.214751+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":664,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-03-30T13:06:14.248918+0000","last_change":"2024-03-30T13:05:07.446715+0000","last_active":"2024-03-30T13:06:14.248918+0000","last_peered":"2024-03-30T13:06:14.248918+0000","last_clean":"2024-03-30T13:06:14.248918+0000","last_became_active":"2024-03-30T13:05:07.445475+0000","last_became_peered":"2024-03-30T13:05:07.445475+0000","last_unstale":"2024-03-30T13:06:14.248918+0000","last_undegraded":"2024-03-30T13:06:14.248918+0000","last_fullsized":"2024-03-30T13:06:14.248918+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:05:00.265024+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:05:00.265024+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:17.234152+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":692,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-03-30T13:06:14.249008+0000","last_change":"2024-03-30T13:06:05.397483+0000","last_active":"2024-03-30T13:06:14.249008+0000","last_peered":"2024-03-30T13:06:14.249008+0000","last_clean":"2024-03-30T13:06:14.249008+0000","last_became_active":"2024-03-30T13:06:03.206037+0000","last_became_peered":"2024-03-30T13:06:03.206037+0000","last_unstale":"2024-03-30T13:06:14.249008+0000","last_undegraded":"2024-03-30T13:06:14.249008+0000","last_fullsized":"2024-03-30T13:06:14.249008+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:06:05.397104+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:06:05.397104+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:28.298853+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":49,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.795606+0000","last_change":"2024-03-30T13:06:03.317478+0000","last_active":"2024-03-30T13:06:17.795606+0000","last_peered":"2024-03-30T13:06:17.795606+0000","last_clean":"2024-03-30T13:06:17.795606+0000","last_became_active":"2024-03-30T13:06:03.194120+0000","last_became_peered":"2024-03-30T13:06:03.194120+0000","last_unstale":"2024-03-30T13:06:17.795606+0000","last_undegraded":"2024-03-30T13:06:17.795606+0000","last_fullsized":"2024-03-30T13:06:17.795606+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:05:28.484860+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_clean_scrub_stamp":"2024-03-30T13:05:28.484860+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:43.514196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":202,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-03-30T13:06:14.045587+0000","last_change":"2024-03-30T13:05:55.776927+0000","last_active":"2024-03-30T13:06:14.045587+0000","last_peered":"2024-03-30T13:06:14.045587+0000","last_clean":"2024-03-30T13:06:14.045587+0000","last_became_active":"2024-03-30T13:03:25.178234+0000","last_became_peered":"2024-03-30T13:03:25.178234+0000","last_unstale":"2024-03-30T13:06:14.045587+0000","last_undegraded":"2024-03-30T13:06:14.045587+0000","last_fullsized":"2024-03-30T13:06:14.045587+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:05:55.776693+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:05:55.776693+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:59.319019+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":49,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.795663+0000","last_change":"2024-03-30T13:06:03.294781+0000","last_active":"2024-03-30T13:06:17.795663+0000","last_peered":"2024-03-30T13:06:17.795663+0000","last_clean":"2024-03-30T13:06:17.795663+0000","last_became_active":"2024-03-30T13:06:03.198523+0000","last_became_peered":"2024-03-30T13:06:03.198523+0000","last_unstale":"2024-03-30T13:06:17.795663+0000","last_undegraded":"2024-03-30T13:06:17.795663+0000","last_fullsized":"2024-03-30T13:06:17.795663+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:05:22.298091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:05:22.298091+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:50.590330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":652,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.328560+0000","last_change":"2024-03-30T13:06:03.289927+0000","last_active":"2024-03-30T13:06:17.328560+0000","last_peered":"2024-03-30T13:06:17.328560+0000","last_clean":"2024-03-30T13:06:17.328560+0000","last_became_active":"2024-03-30T13:06:03.206122+0000","last_became_peered":"2024-03-30T13:06:03.206122+0000","last_unstale":"2024-03-30T13:06:17.328560+0000","last_undegraded":"2024-03-30T13:06:17.328560+0000","last_fullsized":"2024-03-30T13:06:17.328560+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:55.118814+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:04:55.118814+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:23.842135+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":20907644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":305278190,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":227,"seq":974957576218,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50196,"kb_used_data":18504,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321644,"statfs":{"total":96636764160,"available":96585363456,"internally_reserved":0,"allocated":18948096,"data_stored":17025934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":274,"seq":1176821039110,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55432,"kb_used_data":23284,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94316408,"statfs":{"total":96636764160,"available":96580001792,"internally_reserved":0,"allocated":23842816,"data_stored":21900251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6053,"internal_metadata":32893019},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824016,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47568,"kb_used_data":14976,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94324272,"statfs":{"total":96636764160,"available":96588054528,"internally_reserved":0,"allocated":15335424,"data_stored":13399660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509572,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40916,"kb_used_data":9732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330924,"statfs":{"total":96636764160,"available":96594866176,"internally_reserved":0,"allocated":9965568,"data_stored":8036605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285658,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40292,"kb_used_data":8600,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94331548,"statfs":{"total":96636764160,"available":96595505152,"internally_reserved":0,"allocated":8806400,"data_stored":6863835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509572,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48084,"kb_used_data":16896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323756,"statfs":{"total":96636764160,"available":96587526144,"internally_reserved":0,"allocated":17301504,"data_stored":15379342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":206,"seq":884763263013,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46468,"kb_used_data":13428,"kb_used_omap":9,"kb_used_meta":33014,"kb_avail":94325372,"statfs":{"total":96636764160,"available":96589180928,"internally_reserved":0,"allocated":13750272,"data_stored":11834911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9338,"internal_metadata":33807238},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574981,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47120,"kb_used_data":15932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324720,"statfs":{"total":96636764160,"available":96588513280,"internally_reserved":0,"allocated":16314368,"data_stored":14371803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:06:19.026 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:06:19.027 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:06:19.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:19.138+0000 7fbfbf082640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:19.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:19.154+0000 7fbfbf082640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:19.327 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:06:19.469 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":292,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:06:18.331664+0000","last_up_change":"2024-03-30T13:05:58.131737+0000","last_in_change":"2024-03-30T13:06:02.169330+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":17,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":77,"up_from":83,"up_thru":276,"down_at":78,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6818","nonce":3104224893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6821","nonce":3104224893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6828","nonce":3104224893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6825","nonce":3104224893}]},"public_addr":"172.21.15.67:6818/3104224893","cluster_addr":"172.21.15.67:6821/3104224893","heartbeat_back_addr":"172.21.15.67:6828/3104224893","heartbeat_front_addr":"172.21.15.67:6825/3104224893","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":267,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":276,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6801","nonce":1504348724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6803","nonce":1504348724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6807","nonce":1504348724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6805","nonce":1504348724}]},"public_addr":"172.21.15.67:6801/1504348724","cluster_addr":"172.21.15.67:6803/1504348724","heartbeat_back_addr":"172.21.15.67:6807/1504348724","heartbeat_front_addr":"172.21.15.67:6805/1504348724","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":135,"last_clean_end":243,"up_from":250,"up_thru":268,"down_at":244,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6803","nonce":3986522908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6807","nonce":3986522908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6815","nonce":3986522908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6811","nonce":3986522908}]},"public_addr":"172.21.15.82:6803/3986522908","cluster_addr":"172.21.15.82:6807/3986522908","heartbeat_back_addr":"172.21.15.82:6815/3986522908","heartbeat_front_addr":"172.21.15.82:6811/3986522908","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":156,"last_clean_end":266,"up_from":274,"up_thru":276,"down_at":267,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2286739047},{"type":"v1","addr":"172.21.15.82:6817","nonce":2286739047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2286739047},{"type":"v1","addr":"172.21.15.82:6819","nonce":2286739047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2286739047},{"type":"v1","addr":"172.21.15.82:6823","nonce":2286739047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2286739047},{"type":"v1","addr":"172.21.15.82:6821","nonce":2286739047}]},"public_addr":"172.21.15.82:6817/2286739047","cluster_addr":"172.21.15.82:6819/2286739047","heartbeat_back_addr":"172.21.15.82:6823/2286739047","heartbeat_front_addr":"172.21.15.82:6821/2286739047","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":222,"up_from":227,"up_thru":276,"down_at":223,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3068584906},{"type":"v1","addr":"172.21.15.82:6801","nonce":3068584906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3068584906},{"type":"v1","addr":"172.21.15.82:6806","nonce":3068584906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3068584906},{"type":"v1","addr":"172.21.15.82:6814","nonce":3068584906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3068584906},{"type":"v1","addr":"172.21.15.82:6810","nonce":3068584906}]},"public_addr":"172.21.15.82:6801/3068584906","cluster_addr":"172.21.15.82:6806/3068584906","heartbeat_back_addr":"172.21.15.82:6814/3068584906","heartbeat_front_addr":"172.21.15.82:6810/3068584906","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":180,"last_clean_end":197,"up_from":206,"up_thru":250,"down_at":198,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":1241244807},{"type":"v1","addr":"172.21.15.82:6825","nonce":1241244807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":1241244807},{"type":"v1","addr":"172.21.15.82:6827","nonce":1241244807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":1241244807},{"type":"v1","addr":"172.21.15.82:6831","nonce":1241244807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":1241244807},{"type":"v1","addr":"172.21.15.82:6829","nonce":1241244807}]},"public_addr":"172.21.15.82:6825/1241244807","cluster_addr":"172.21.15.82:6827/1241244807","heartbeat_back_addr":"172.21.15.82:6831/1241244807","heartbeat_front_addr":"172.21.15.82:6829/1241244807","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:05:51.345150+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:05:48.558676+0000","dead_epoch":78},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:05:25.564709+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:06:09.409172+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:05:33.245001+0000","dead_epoch":244},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:05:56.779192+0000","dead_epoch":267},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:05:26.630000+0000","dead_epoch":223},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:05:16.544575+0000","dead_epoch":198}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:06:19.470 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:06:19.806 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:06:19.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:19.934+0000 7fbb52cb8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:19.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:19.954+0000 7fbb52cb8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:20.126 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:06:20.146 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:06:20.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:20.266+0000 7f7f237b9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:20.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:20.282+0000 7f7f237b9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:20.463 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:06:20.463 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:06:20.482 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:06:20.482 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:06:20.482 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:06:20.483 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:06:20.483 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 3, 1, 7, 6, 4, 5] 2024-03-30T13:06:20.483 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:06:20.483 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:06:20.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:20.598+0000 7f6186b07640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:20.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:20.610+0000 7f6186b07640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:20.783 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:06:20.784 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:06:20.800 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":609,"stamp":"2024-03-30T13:06:19.966220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":305278190,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":376204,"kb_used_data":121496,"kb_used_omap":35,"kb_used_meta":254556,"kb_avail":754598516,"statfs":{"total":773094113280,"available":772708880384,"internally_reserved":0,"allocated":124411904,"data_stored":108878185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36669,"internal_metadata":260665539},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.662289"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":642,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-03-30T13:06:14.249041+0000","last_change":"2024-03-30T13:06:03.258135+0000","last_active":"2024-03-30T13:06:14.249041+0000","last_peered":"2024-03-30T13:06:14.249041+0000","last_clean":"2024-03-30T13:06:14.249041+0000","last_became_active":"2024-03-30T13:06:03.198084+0000","last_became_peered":"2024-03-30T13:06:03.198084+0000","last_unstale":"2024-03-30T13:06:14.249041+0000","last_undegraded":"2024-03-30T13:06:14.249041+0000","last_fullsized":"2024-03-30T13:06:14.249041+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:05:31.459608+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:05:31.459608+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:33.232176+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18352980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":595,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-03-30T13:06:16.317673+0000","last_change":"2024-03-30T13:06:14.050179+0000","last_active":"2024-03-30T13:06:16.317673+0000","last_peered":"2024-03-30T13:06:16.317673+0000","last_clean":"2024-03-30T13:06:16.317673+0000","last_became_active":"2024-03-30T13:05:07.444962+0000","last_became_peered":"2024-03-30T13:05:07.444962+0000","last_unstale":"2024-03-30T13:06:16.317673+0000","last_undegraded":"2024-03-30T13:06:16.317673+0000","last_fullsized":"2024-03-30T13:06:16.317673+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:06:14.049898+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:06:14.049898+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:39.577970+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":232,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.796317+0000","last_change":"2024-03-30T13:06:11.665355+0000","last_active":"2024-03-30T13:06:17.796317+0000","last_peered":"2024-03-30T13:06:17.796317+0000","last_clean":"2024-03-30T13:06:17.796317+0000","last_became_active":"2024-03-30T13:05:07.442521+0000","last_became_peered":"2024-03-30T13:05:07.442521+0000","last_unstale":"2024-03-30T13:06:17.796317+0000","last_undegraded":"2024-03-30T13:06:17.796317+0000","last_fullsized":"2024-03-30T13:06:17.796317+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:06:11.665099+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_clean_scrub_stamp":"2024-03-30T13:06:11.665099+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:34.778370+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":229,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.795879+0000","last_change":"2024-03-30T13:06:06.768447+0000","last_active":"2024-03-30T13:06:17.795879+0000","last_peered":"2024-03-30T13:06:17.795879+0000","last_clean":"2024-03-30T13:06:17.795879+0000","last_became_active":"2024-03-30T13:04:16.039346+0000","last_became_peered":"2024-03-30T13:04:16.039346+0000","last_unstale":"2024-03-30T13:06:17.795879+0000","last_undegraded":"2024-03-30T13:06:17.795879+0000","last_fullsized":"2024-03-30T13:06:17.795879+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:06:06.768191+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:06:06.768191+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:34.667227+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":43,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.795948+0000","last_change":"2024-03-30T13:06:03.311279+0000","last_active":"2024-03-30T13:06:17.795948+0000","last_peered":"2024-03-30T13:06:17.795948+0000","last_clean":"2024-03-30T13:06:17.795948+0000","last_became_active":"2024-03-30T13:06:03.203196+0000","last_became_peered":"2024-03-30T13:06:03.203196+0000","last_unstale":"2024-03-30T13:06:17.795948+0000","last_undegraded":"2024-03-30T13:06:17.795948+0000","last_fullsized":"2024-03-30T13:06:17.795948+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:05:17.370216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:05:17.370216+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:28.258191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":686,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.777471+0000","last_change":"2024-03-30T13:06:03.664841+0000","last_active":"2024-03-30T13:06:17.777471+0000","last_peered":"2024-03-30T13:06:17.777471+0000","last_clean":"2024-03-30T13:06:17.777471+0000","last_became_active":"2024-03-30T13:06:03.202467+0000","last_became_peered":"2024-03-30T13:06:03.202467+0000","last_unstale":"2024-03-30T13:06:17.777471+0000","last_undegraded":"2024-03-30T13:06:17.777471+0000","last_fullsized":"2024-03-30T13:06:17.777471+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:05:13.431611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:05:13.431611+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:36.582588+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":45579924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":257,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.796620+0000","last_change":"2024-03-30T13:05:54.746839+0000","last_active":"2024-03-30T13:06:17.796620+0000","last_peered":"2024-03-30T13:06:17.796620+0000","last_clean":"2024-03-30T13:06:17.796620+0000","last_became_active":"2024-03-30T13:05:07.440390+0000","last_became_peered":"2024-03-30T13:05:07.440390+0000","last_unstale":"2024-03-30T13:06:17.796620+0000","last_undegraded":"2024-03-30T13:06:17.796620+0000","last_fullsized":"2024-03-30T13:06:17.796620+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:05:54.746578+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:05:54.746578+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:11.361732+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":690,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.328432+0000","last_change":"2024-03-30T13:06:03.273364+0000","last_active":"2024-03-30T13:06:17.328432+0000","last_peered":"2024-03-30T13:06:17.328432+0000","last_clean":"2024-03-30T13:06:17.328432+0000","last_became_active":"2024-03-30T13:06:03.206177+0000","last_became_peered":"2024-03-30T13:06:03.206177+0000","last_unstale":"2024-03-30T13:06:17.328432+0000","last_undegraded":"2024-03-30T13:06:17.328432+0000","last_fullsized":"2024-03-30T13:06:17.328432+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:05:24.040489+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:25.404654+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":753,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.777475+0000","last_change":"2024-03-30T13:06:03.665679+0000","last_active":"2024-03-30T13:06:17.777475+0000","last_peered":"2024-03-30T13:06:17.777475+0000","last_clean":"2024-03-30T13:06:17.777475+0000","last_became_active":"2024-03-30T13:06:03.219523+0000","last_became_peered":"2024-03-30T13:06:03.219523+0000","last_unstale":"2024-03-30T13:06:17.777475+0000","last_undegraded":"2024-03-30T13:06:17.777475+0000","last_fullsized":"2024-03-30T13:06:17.777475+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:05:45.457140+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:40.339270+0000","last_clean_scrub_stamp":"2024-03-30T13:05:45.457140+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:08.613970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":55832128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":513,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-03-30T13:06:16.317638+0000","last_change":"2024-03-30T13:05:12.144999+0000","last_active":"2024-03-30T13:06:16.317638+0000","last_peered":"2024-03-30T13:06:16.317638+0000","last_clean":"2024-03-30T13:06:16.317638+0000","last_became_active":"2024-03-30T13:05:07.445793+0000","last_became_peered":"2024-03-30T13:05:07.445793+0000","last_unstale":"2024-03-30T13:06:16.317638+0000","last_undegraded":"2024-03-30T13:06:16.317638+0000","last_fullsized":"2024-03-30T13:06:16.317638+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:05:12.144734+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:05:12.144734+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:30.214751+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":664,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-03-30T13:06:14.248918+0000","last_change":"2024-03-30T13:05:07.446715+0000","last_active":"2024-03-30T13:06:14.248918+0000","last_peered":"2024-03-30T13:06:14.248918+0000","last_clean":"2024-03-30T13:06:14.248918+0000","last_became_active":"2024-03-30T13:05:07.445475+0000","last_became_peered":"2024-03-30T13:05:07.445475+0000","last_unstale":"2024-03-30T13:06:14.248918+0000","last_undegraded":"2024-03-30T13:06:14.248918+0000","last_fullsized":"2024-03-30T13:06:14.248918+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:05:00.265024+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:05:00.265024+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:17.234152+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":692,"reported_epoch":287,"state":"active+clean","last_fresh":"2024-03-30T13:06:14.249008+0000","last_change":"2024-03-30T13:06:05.397483+0000","last_active":"2024-03-30T13:06:14.249008+0000","last_peered":"2024-03-30T13:06:14.249008+0000","last_clean":"2024-03-30T13:06:14.249008+0000","last_became_active":"2024-03-30T13:06:03.206037+0000","last_became_peered":"2024-03-30T13:06:03.206037+0000","last_unstale":"2024-03-30T13:06:14.249008+0000","last_undegraded":"2024-03-30T13:06:14.249008+0000","last_fullsized":"2024-03-30T13:06:14.249008+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:06:05.397104+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:06:05.397104+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:28.298853+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":49,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.795606+0000","last_change":"2024-03-30T13:06:03.317478+0000","last_active":"2024-03-30T13:06:17.795606+0000","last_peered":"2024-03-30T13:06:17.795606+0000","last_clean":"2024-03-30T13:06:17.795606+0000","last_became_active":"2024-03-30T13:06:03.194120+0000","last_became_peered":"2024-03-30T13:06:03.194120+0000","last_unstale":"2024-03-30T13:06:17.795606+0000","last_undegraded":"2024-03-30T13:06:17.795606+0000","last_fullsized":"2024-03-30T13:06:17.795606+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:05:28.484860+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_clean_scrub_stamp":"2024-03-30T13:05:28.484860+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:43.514196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":209,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.795806+0000","last_change":"2024-03-30T13:05:55.776927+0000","last_active":"2024-03-30T13:06:17.795806+0000","last_peered":"2024-03-30T13:06:17.795806+0000","last_clean":"2024-03-30T13:06:17.795806+0000","last_became_active":"2024-03-30T13:03:25.178234+0000","last_became_peered":"2024-03-30T13:03:25.178234+0000","last_unstale":"2024-03-30T13:06:17.795806+0000","last_undegraded":"2024-03-30T13:06:17.795806+0000","last_fullsized":"2024-03-30T13:06:17.795806+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:05:55.776693+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:05:55.776693+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:59.319019+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":49,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.795663+0000","last_change":"2024-03-30T13:06:03.294781+0000","last_active":"2024-03-30T13:06:17.795663+0000","last_peered":"2024-03-30T13:06:17.795663+0000","last_clean":"2024-03-30T13:06:17.795663+0000","last_became_active":"2024-03-30T13:06:03.198523+0000","last_became_peered":"2024-03-30T13:06:03.198523+0000","last_unstale":"2024-03-30T13:06:17.795663+0000","last_undegraded":"2024-03-30T13:06:17.795663+0000","last_fullsized":"2024-03-30T13:06:17.795663+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:05:22.298091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:05:22.298091+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:50.590330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":652,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.328560+0000","last_change":"2024-03-30T13:06:03.289927+0000","last_active":"2024-03-30T13:06:17.328560+0000","last_peered":"2024-03-30T13:06:17.328560+0000","last_clean":"2024-03-30T13:06:17.328560+0000","last_became_active":"2024-03-30T13:06:03.206122+0000","last_became_peered":"2024-03-30T13:06:03.206122+0000","last_unstale":"2024-03-30T13:06:17.328560+0000","last_undegraded":"2024-03-30T13:06:17.328560+0000","last_fullsized":"2024-03-30T13:06:17.328560+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:55.118814+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:04:55.118814+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:23.842135+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":20907644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":305278190,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":227,"seq":974957576218,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50196,"kb_used_data":18504,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321644,"statfs":{"total":96636764160,"available":96585363456,"internally_reserved":0,"allocated":18948096,"data_stored":17025934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":274,"seq":1176821039110,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55432,"kb_used_data":23284,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94316408,"statfs":{"total":96636764160,"available":96580001792,"internally_reserved":0,"allocated":23842816,"data_stored":21900251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6053,"internal_metadata":32893019},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824016,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47568,"kb_used_data":14976,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94324272,"statfs":{"total":96636764160,"available":96588054528,"internally_reserved":0,"allocated":15335424,"data_stored":13399660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509573,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40948,"kb_used_data":9768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330892,"statfs":{"total":96636764160,"available":96594833408,"internally_reserved":0,"allocated":10002432,"data_stored":8053066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285658,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40292,"kb_used_data":8600,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94331548,"statfs":{"total":96636764160,"available":96595505152,"internally_reserved":0,"allocated":8806400,"data_stored":6863835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509573,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48124,"kb_used_data":16944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323716,"statfs":{"total":96636764160,"available":96587485184,"internally_reserved":0,"allocated":17350656,"data_stored":15401290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":206,"seq":884763263014,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46524,"kb_used_data":13488,"kb_used_omap":9,"kb_used_meta":33014,"kb_avail":94325316,"statfs":{"total":96636764160,"available":96589123584,"internally_reserved":0,"allocated":13811712,"data_stored":11862346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9338,"internal_metadata":33807238},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574981,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47120,"kb_used_data":15932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324720,"statfs":{"total":96636764160,"available":96588513280,"internally_reserved":0,"allocated":16314368,"data_stored":14371803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:06:20.802 INFO:tasks.thrashosds.thrasher:Choosing PG 1.9 with acting set [7, 3, 2] 2024-03-30T13:06:20.802 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 2, 3, 1, 7, 6, 4, 5] 2024-03-30T13:06:20.802 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-30T13:06:20.802 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:06:20.865 INFO:tasks.ceph.osd.7:Stopped 2024-03-30T13:06:20.865 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 2, 3, 1, 7, 6, 4, 5] 2024-03-30T13:06:20.865 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 7 2024-03-30T13:06:20.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:20.986+0000 7fc297bc9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:20.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:21.002+0000 7fc297bc9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:21.176 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:06:21.178+0000 7fbb38f08640 -1 mon.a@0(leader).osd e292 definitely_dead 0 2024-03-30T13:06:21.365 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.7. 2024-03-30T13:06:21.389 INFO:tasks.thrashosds.thrasher:dead_osds=[7], live_osds=[0, 2, 3, 1, 6, 4, 5] 2024-03-30T13:06:21.389 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:06:21.389 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:06:21.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:21.502+0000 7f493bbe7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:21.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:21.514+0000 7f493bbe7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:21.680 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:06:21.681 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:06:21.697 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":610,"stamp":"2024-03-30T13:06:21.363146+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":305278190,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":329744,"kb_used_data":108068,"kb_used_omap":26,"kb_used_meta":221541,"kb_avail":660273136,"statfs":{"total":676457349120,"available":676119691264,"internally_reserved":0,"allocated":110661632,"data_stored":97043274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27331,"internal_metadata":226858301},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.397912"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":650,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-03-30T13:06:18.343438+0000","last_change":"2024-03-30T13:06:03.258135+0000","last_active":"2024-03-30T13:06:18.343438+0000","last_peered":"2024-03-30T13:06:18.343438+0000","last_clean":"2024-03-30T13:06:18.343438+0000","last_became_active":"2024-03-30T13:06:03.198084+0000","last_became_peered":"2024-03-30T13:06:03.198084+0000","last_unstale":"2024-03-30T13:06:18.343438+0000","last_undegraded":"2024-03-30T13:06:18.343438+0000","last_fullsized":"2024-03-30T13:06:18.343438+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:05:31.459608+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:05:31.459608+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:33.232176+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18352980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":595,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-03-30T13:06:16.317673+0000","last_change":"2024-03-30T13:06:14.050179+0000","last_active":"2024-03-30T13:06:16.317673+0000","last_peered":"2024-03-30T13:06:16.317673+0000","last_clean":"2024-03-30T13:06:16.317673+0000","last_became_active":"2024-03-30T13:05:07.444962+0000","last_became_peered":"2024-03-30T13:05:07.444962+0000","last_unstale":"2024-03-30T13:06:16.317673+0000","last_undegraded":"2024-03-30T13:06:16.317673+0000","last_fullsized":"2024-03-30T13:06:16.317673+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:06:14.049898+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:06:14.049898+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:39.577970+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":232,"reported_epoch":291,"state":"stale+active+clean","last_fresh":"2024-03-30T13:06:17.796317+0000","last_change":"2024-03-30T13:06:11.665355+0000","last_active":"2024-03-30T13:06:17.796317+0000","last_peered":"2024-03-30T13:06:17.796317+0000","last_clean":"2024-03-30T13:06:17.796317+0000","last_became_active":"2024-03-30T13:05:07.442521+0000","last_became_peered":"2024-03-30T13:05:07.442521+0000","last_unstale":"2024-03-30T13:06:21.363168+0000","last_undegraded":"2024-03-30T13:06:17.796317+0000","last_fullsized":"2024-03-30T13:06:17.796317+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:06:11.665099+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_clean_scrub_stamp":"2024-03-30T13:06:11.665099+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:34.778370+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":229,"reported_epoch":291,"state":"stale+active+clean","last_fresh":"2024-03-30T13:06:17.795879+0000","last_change":"2024-03-30T13:06:06.768447+0000","last_active":"2024-03-30T13:06:17.795879+0000","last_peered":"2024-03-30T13:06:17.795879+0000","last_clean":"2024-03-30T13:06:17.795879+0000","last_became_active":"2024-03-30T13:04:16.039346+0000","last_became_peered":"2024-03-30T13:04:16.039346+0000","last_unstale":"2024-03-30T13:06:21.363172+0000","last_undegraded":"2024-03-30T13:06:17.795879+0000","last_fullsized":"2024-03-30T13:06:17.795879+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:06:06.768191+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:06:06.768191+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:34.667227+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":43,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.795948+0000","last_change":"2024-03-30T13:06:03.311279+0000","last_active":"2024-03-30T13:06:17.795948+0000","last_peered":"2024-03-30T13:06:17.795948+0000","last_clean":"2024-03-30T13:06:17.795948+0000","last_became_active":"2024-03-30T13:06:03.203196+0000","last_became_peered":"2024-03-30T13:06:03.203196+0000","last_unstale":"2024-03-30T13:06:17.795948+0000","last_undegraded":"2024-03-30T13:06:17.795948+0000","last_fullsized":"2024-03-30T13:06:17.795948+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:05:17.370216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:05:17.370216+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:28.258191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":686,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.777471+0000","last_change":"2024-03-30T13:06:03.664841+0000","last_active":"2024-03-30T13:06:17.777471+0000","last_peered":"2024-03-30T13:06:17.777471+0000","last_clean":"2024-03-30T13:06:17.777471+0000","last_became_active":"2024-03-30T13:06:03.202467+0000","last_became_peered":"2024-03-30T13:06:03.202467+0000","last_unstale":"2024-03-30T13:06:17.777471+0000","last_undegraded":"2024-03-30T13:06:17.777471+0000","last_fullsized":"2024-03-30T13:06:17.777471+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:05:13.431611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:05:13.431611+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:36.582588+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":45579924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":257,"reported_epoch":291,"state":"stale+active+clean","last_fresh":"2024-03-30T13:06:17.796620+0000","last_change":"2024-03-30T13:05:54.746839+0000","last_active":"2024-03-30T13:06:17.796620+0000","last_peered":"2024-03-30T13:06:17.796620+0000","last_clean":"2024-03-30T13:06:17.796620+0000","last_became_active":"2024-03-30T13:05:07.440390+0000","last_became_peered":"2024-03-30T13:05:07.440390+0000","last_unstale":"2024-03-30T13:06:21.363174+0000","last_undegraded":"2024-03-30T13:06:17.796620+0000","last_fullsized":"2024-03-30T13:06:17.796620+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:05:54.746578+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:05:54.746578+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:11.361732+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":690,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.328432+0000","last_change":"2024-03-30T13:06:03.273364+0000","last_active":"2024-03-30T13:06:17.328432+0000","last_peered":"2024-03-30T13:06:17.328432+0000","last_clean":"2024-03-30T13:06:17.328432+0000","last_became_active":"2024-03-30T13:06:03.206177+0000","last_became_peered":"2024-03-30T13:06:03.206177+0000","last_unstale":"2024-03-30T13:06:17.328432+0000","last_undegraded":"2024-03-30T13:06:17.328432+0000","last_fullsized":"2024-03-30T13:06:17.328432+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:05:24.040489+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:25.404654+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":753,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.777475+0000","last_change":"2024-03-30T13:06:03.665679+0000","last_active":"2024-03-30T13:06:17.777475+0000","last_peered":"2024-03-30T13:06:17.777475+0000","last_clean":"2024-03-30T13:06:17.777475+0000","last_became_active":"2024-03-30T13:06:03.219523+0000","last_became_peered":"2024-03-30T13:06:03.219523+0000","last_unstale":"2024-03-30T13:06:17.777475+0000","last_undegraded":"2024-03-30T13:06:17.777475+0000","last_fullsized":"2024-03-30T13:06:17.777475+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:05:45.457140+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:40.339270+0000","last_clean_scrub_stamp":"2024-03-30T13:05:45.457140+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:08.613970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":55832128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":513,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-03-30T13:06:16.317638+0000","last_change":"2024-03-30T13:05:12.144999+0000","last_active":"2024-03-30T13:06:16.317638+0000","last_peered":"2024-03-30T13:06:16.317638+0000","last_clean":"2024-03-30T13:06:16.317638+0000","last_became_active":"2024-03-30T13:05:07.445793+0000","last_became_peered":"2024-03-30T13:05:07.445793+0000","last_unstale":"2024-03-30T13:06:16.317638+0000","last_undegraded":"2024-03-30T13:06:16.317638+0000","last_fullsized":"2024-03-30T13:06:16.317638+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:05:12.144734+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:05:12.144734+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:30.214751+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":681,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-03-30T13:06:19.487398+0000","last_change":"2024-03-30T13:06:19.487398+0000","last_active":"2024-03-30T13:06:19.487398+0000","last_peered":"2024-03-30T13:06:19.487398+0000","last_clean":"2024-03-30T13:06:19.487398+0000","last_became_active":"2024-03-30T13:05:07.445475+0000","last_became_peered":"2024-03-30T13:05:07.445475+0000","last_unstale":"2024-03-30T13:06:19.487398+0000","last_undegraded":"2024-03-30T13:06:19.487398+0000","last_fullsized":"2024-03-30T13:06:19.487398+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:06:19.487011+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:06:19.487011+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:46.181120+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":700,"reported_epoch":292,"state":"active+clean","last_fresh":"2024-03-30T13:06:18.343198+0000","last_change":"2024-03-30T13:06:05.397483+0000","last_active":"2024-03-30T13:06:18.343198+0000","last_peered":"2024-03-30T13:06:18.343198+0000","last_clean":"2024-03-30T13:06:18.343198+0000","last_became_active":"2024-03-30T13:06:03.206037+0000","last_became_peered":"2024-03-30T13:06:03.206037+0000","last_unstale":"2024-03-30T13:06:18.343198+0000","last_undegraded":"2024-03-30T13:06:18.343198+0000","last_fullsized":"2024-03-30T13:06:18.343198+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:06:05.397104+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:06:05.397104+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:28.298853+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":49,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.795606+0000","last_change":"2024-03-30T13:06:03.317478+0000","last_active":"2024-03-30T13:06:17.795606+0000","last_peered":"2024-03-30T13:06:17.795606+0000","last_clean":"2024-03-30T13:06:17.795606+0000","last_became_active":"2024-03-30T13:06:03.194120+0000","last_became_peered":"2024-03-30T13:06:03.194120+0000","last_unstale":"2024-03-30T13:06:17.795606+0000","last_undegraded":"2024-03-30T13:06:17.795606+0000","last_fullsized":"2024-03-30T13:06:17.795606+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:05:28.484860+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_clean_scrub_stamp":"2024-03-30T13:05:28.484860+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:43.514196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":209,"reported_epoch":291,"state":"stale+active+clean","last_fresh":"2024-03-30T13:06:17.795806+0000","last_change":"2024-03-30T13:05:55.776927+0000","last_active":"2024-03-30T13:06:17.795806+0000","last_peered":"2024-03-30T13:06:17.795806+0000","last_clean":"2024-03-30T13:06:17.795806+0000","last_became_active":"2024-03-30T13:03:25.178234+0000","last_became_peered":"2024-03-30T13:03:25.178234+0000","last_unstale":"2024-03-30T13:06:21.363178+0000","last_undegraded":"2024-03-30T13:06:17.795806+0000","last_fullsized":"2024-03-30T13:06:17.795806+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:05:55.776693+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:05:55.776693+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:59.319019+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":49,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.795663+0000","last_change":"2024-03-30T13:06:03.294781+0000","last_active":"2024-03-30T13:06:17.795663+0000","last_peered":"2024-03-30T13:06:17.795663+0000","last_clean":"2024-03-30T13:06:17.795663+0000","last_became_active":"2024-03-30T13:06:03.198523+0000","last_became_peered":"2024-03-30T13:06:03.198523+0000","last_unstale":"2024-03-30T13:06:17.795663+0000","last_undegraded":"2024-03-30T13:06:17.795663+0000","last_fullsized":"2024-03-30T13:06:17.795663+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:05:22.298091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:05:22.298091+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:50.590330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":652,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-03-30T13:06:17.328560+0000","last_change":"2024-03-30T13:06:03.289927+0000","last_active":"2024-03-30T13:06:17.328560+0000","last_peered":"2024-03-30T13:06:17.328560+0000","last_clean":"2024-03-30T13:06:17.328560+0000","last_became_active":"2024-03-30T13:06:03.206122+0000","last_became_peered":"2024-03-30T13:06:03.206122+0000","last_unstale":"2024-03-30T13:06:17.328560+0000","last_undegraded":"2024-03-30T13:06:17.328560+0000","last_fullsized":"2024-03-30T13:06:17.328560+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:04:55.118814+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:04:55.118814+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:23.842135+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":20907644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":305278190,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":227,"seq":974957576219,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50260,"kb_used_data":18564,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321580,"statfs":{"total":96636764160,"available":96585297920,"internally_reserved":0,"allocated":19009536,"data_stored":17053369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":274,"seq":1176821039110,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55432,"kb_used_data":23284,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94316408,"statfs":{"total":96636764160,"available":96580001792,"internally_reserved":0,"allocated":23842816,"data_stored":21900251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6053,"internal_metadata":32893019},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824016,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47568,"kb_used_data":14976,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94324272,"statfs":{"total":96636764160,"available":96588054528,"internally_reserved":0,"allocated":15335424,"data_stored":13399660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509573,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40948,"kb_used_data":9768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330892,"statfs":{"total":96636764160,"available":96594833408,"internally_reserved":0,"allocated":10002432,"data_stored":8053066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285658,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40292,"kb_used_data":8600,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94331548,"statfs":{"total":96636764160,"available":96595505152,"internally_reserved":0,"allocated":8806400,"data_stored":6863835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509573,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48124,"kb_used_data":16944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323716,"statfs":{"total":96636764160,"available":96587485184,"internally_reserved":0,"allocated":17350656,"data_stored":15401290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574981,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47120,"kb_used_data":15932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324720,"statfs":{"total":96636764160,"available":96588513280,"internally_reserved":0,"allocated":16314368,"data_stored":14371803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:06:21.699 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:06:46.699 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:06:46.700 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:06:46.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:46.830+0000 7f05f5496640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:46.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:46.842+0000 7f05f5496640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:47.017 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:06:47.017 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:06:47.035 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":626,"stamp":"2024-03-30T13:06:45.975182+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":300858015,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":340084,"kb_used_data":118448,"kb_used_omap":26,"kb_used_meta":221541,"kb_avail":660262796,"statfs":{"total":676457349120,"available":676109103104,"internally_reserved":0,"allocated":121290752,"data_stored":107483604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":27331,"internal_metadata":226858301},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002145"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":665,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-30T13:06:43.429900+0000","last_change":"2024-03-30T13:06:34.434294+0000","last_active":"2024-03-30T13:06:43.429900+0000","last_peered":"2024-03-30T13:06:43.429900+0000","last_clean":"2024-03-30T13:06:43.429900+0000","last_became_active":"2024-03-30T13:06:03.198084+0000","last_became_peered":"2024-03-30T13:06:03.198084+0000","last_unstale":"2024-03-30T13:06:43.429900+0000","last_undegraded":"2024-03-30T13:06:43.429900+0000","last_fullsized":"2024-03-30T13:06:43.429900+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:06:34.434031+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:06:34.434031+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:37.346053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18352980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":613,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-03-30T13:06:22.418175+0000","last_change":"2024-03-30T13:06:22.418175+0000","last_active":"2024-03-30T13:06:22.418175+0000","last_peered":"2024-03-30T13:06:22.418175+0000","last_clean":"2024-03-30T13:06:22.418175+0000","last_became_active":"2024-03-30T13:06:22.379351+0000","last_became_peered":"2024-03-30T13:06:22.379351+0000","last_unstale":"2024-03-30T13:06:22.418175+0000","last_undegraded":"2024-03-30T13:06:22.418175+0000","last_fullsized":"2024-03-30T13:06:22.418175+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:06:14.049898+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:06:14.049898+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:27.615716+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":21,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-30T13:06:43.392587+0000","last_change":"2024-03-30T13:06:22.464442+0000","last_active":"2024-03-30T13:06:43.392587+0000","last_peered":"2024-03-30T13:06:43.392587+0000","last_clean":"2024-03-30T13:06:43.392587+0000","last_became_active":"2024-03-30T13:06:22.383868+0000","last_became_peered":"2024-03-30T13:06:22.383868+0000","last_unstale":"2024-03-30T13:06:43.392587+0000","last_undegraded":"2024-03-30T13:06:43.392587+0000","last_fullsized":"2024-03-30T13:06:43.392587+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:06:11.665099+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_clean_scrub_stamp":"2024-03-30T13:06:11.665099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:29.189179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":17,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-03-30T13:06:22.444619+0000","last_change":"2024-03-30T13:06:22.443287+0000","last_active":"2024-03-30T13:06:22.444619+0000","last_peered":"2024-03-30T13:06:22.444619+0000","last_clean":"2024-03-30T13:06:22.444619+0000","last_became_active":"2024-03-30T13:06:22.380569+0000","last_became_peered":"2024-03-30T13:06:22.380569+0000","last_unstale":"2024-03-30T13:06:22.444619+0000","last_undegraded":"2024-03-30T13:06:22.444619+0000","last_fullsized":"2024-03-30T13:06:22.444619+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:06:06.768191+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:06:06.768191+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:11.523480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":59,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-30T13:06:43.124590+0000","last_change":"2024-03-30T13:06:29.679546+0000","last_active":"2024-03-30T13:06:43.124590+0000","last_peered":"2024-03-30T13:06:43.124590+0000","last_clean":"2024-03-30T13:06:43.124590+0000","last_became_active":"2024-03-30T13:06:03.203196+0000","last_became_peered":"2024-03-30T13:06:03.203196+0000","last_unstale":"2024-03-30T13:06:43.124590+0000","last_undegraded":"2024-03-30T13:06:43.124590+0000","last_fullsized":"2024-03-30T13:06:43.124590+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:06:29.679251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:06:29.679251+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:46.794376+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":700,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-03-30T13:06:38.405534+0000","last_change":"2024-03-30T13:06:38.405609+0000","last_active":"2024-03-30T13:06:38.405534+0000","last_peered":"2024-03-30T13:06:38.405534+0000","last_clean":"2024-03-30T13:06:38.405534+0000","last_became_active":"2024-03-30T13:06:03.202467+0000","last_became_peered":"2024-03-30T13:06:03.202467+0000","last_unstale":"2024-03-30T13:06:38.405534+0000","last_undegraded":"2024-03-30T13:06:38.405534+0000","last_fullsized":"2024-03-30T13:06:38.405534+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:06:38.405328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:06:38.405328+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:04.811288+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":45579924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":32,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-03-30T13:06:22.511774+0000","last_change":"2024-03-30T13:06:22.511774+0000","last_active":"2024-03-30T13:06:22.511774+0000","last_peered":"2024-03-30T13:06:22.511774+0000","last_clean":"2024-03-30T13:06:22.511774+0000","last_became_active":"2024-03-30T13:06:22.382410+0000","last_became_peered":"2024-03-30T13:06:22.382410+0000","last_unstale":"2024-03-30T13:06:22.511774+0000","last_undegraded":"2024-03-30T13:06:22.511774+0000","last_fullsized":"2024-03-30T13:06:22.511774+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:05:54.746578+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:05:54.746578+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:05.291303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":704,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-03-30T13:06:29.144628+0000","last_change":"2024-03-30T13:06:29.144702+0000","last_active":"2024-03-30T13:06:29.144628+0000","last_peered":"2024-03-30T13:06:29.144628+0000","last_clean":"2024-03-30T13:06:29.144628+0000","last_became_active":"2024-03-30T13:06:03.206177+0000","last_became_peered":"2024-03-30T13:06:03.206177+0000","last_unstale":"2024-03-30T13:06:29.144628+0000","last_undegraded":"2024-03-30T13:06:29.144628+0000","last_fullsized":"2024-03-30T13:06:29.144628+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:06:29.144448+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:06:29.144448+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:37.166703+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":771,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-03-30T13:06:22.467742+0000","last_change":"2024-03-30T13:06:22.467742+0000","last_active":"2024-03-30T13:06:22.467742+0000","last_peered":"2024-03-30T13:06:22.467742+0000","last_clean":"2024-03-30T13:06:22.467742+0000","last_became_active":"2024-03-30T13:06:22.388048+0000","last_became_peered":"2024-03-30T13:06:22.388048+0000","last_unstale":"2024-03-30T13:06:22.467742+0000","last_undegraded":"2024-03-30T13:06:22.467742+0000","last_fullsized":"2024-03-30T13:06:22.467742+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:05:45.457140+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:40.339270+0000","last_clean_scrub_stamp":"2024-03-30T13:05:45.457140+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:58.873425+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":529,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-03-30T13:06:34.923999+0000","last_change":"2024-03-30T13:06:34.924067+0000","last_active":"2024-03-30T13:06:34.923999+0000","last_peered":"2024-03-30T13:06:34.923999+0000","last_clean":"2024-03-30T13:06:34.923999+0000","last_became_active":"2024-03-30T13:05:07.445793+0000","last_became_peered":"2024-03-30T13:05:07.445793+0000","last_unstale":"2024-03-30T13:06:34.923999+0000","last_undegraded":"2024-03-30T13:06:34.923999+0000","last_fullsized":"2024-03-30T13:06:34.923999+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:06:34.923827+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:06:34.923827+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:01.504665+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":687,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-30T13:06:43.430001+0000","last_change":"2024-03-30T13:06:19.487398+0000","last_active":"2024-03-30T13:06:43.430001+0000","last_peered":"2024-03-30T13:06:43.430001+0000","last_clean":"2024-03-30T13:06:43.430001+0000","last_became_active":"2024-03-30T13:05:07.445475+0000","last_became_peered":"2024-03-30T13:05:07.445475+0000","last_unstale":"2024-03-30T13:06:43.430001+0000","last_undegraded":"2024-03-30T13:06:43.430001+0000","last_fullsized":"2024-03-30T13:06:43.430001+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:06:19.487011+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:06:19.487011+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:46.181120+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":707,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-30T13:06:43.429918+0000","last_change":"2024-03-30T13:06:05.397483+0000","last_active":"2024-03-30T13:06:43.429918+0000","last_peered":"2024-03-30T13:06:43.429918+0000","last_clean":"2024-03-30T13:06:43.429918+0000","last_became_active":"2024-03-30T13:06:03.206037+0000","last_became_peered":"2024-03-30T13:06:03.206037+0000","last_unstale":"2024-03-30T13:06:43.429918+0000","last_undegraded":"2024-03-30T13:06:43.429918+0000","last_fullsized":"2024-03-30T13:06:43.429918+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:06:05.397104+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:06:05.397104+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:28.298853+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":57,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-30T13:06:43.124515+0000","last_change":"2024-03-30T13:06:03.317478+0000","last_active":"2024-03-30T13:06:43.124515+0000","last_peered":"2024-03-30T13:06:43.124515+0000","last_clean":"2024-03-30T13:06:43.124515+0000","last_became_active":"2024-03-30T13:06:03.194120+0000","last_became_peered":"2024-03-30T13:06:03.194120+0000","last_unstale":"2024-03-30T13:06:43.124515+0000","last_undegraded":"2024-03-30T13:06:43.124515+0000","last_fullsized":"2024-03-30T13:06:43.124515+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:05:28.484860+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_clean_scrub_stamp":"2024-03-30T13:05:28.484860+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:43.514196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":17,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-30T13:06:43.124583+0000","last_change":"2024-03-30T13:06:22.413674+0000","last_active":"2024-03-30T13:06:43.124583+0000","last_peered":"2024-03-30T13:06:43.124583+0000","last_clean":"2024-03-30T13:06:43.124583+0000","last_became_active":"2024-03-30T13:06:22.381859+0000","last_became_peered":"2024-03-30T13:06:22.381859+0000","last_unstale":"2024-03-30T13:06:43.124583+0000","last_undegraded":"2024-03-30T13:06:43.124583+0000","last_fullsized":"2024-03-30T13:06:43.124583+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:05:55.776693+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:05:55.776693+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:18.513066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":57,"reported_epoch":295,"state":"active+clean","last_fresh":"2024-03-30T13:06:43.124590+0000","last_change":"2024-03-30T13:06:03.294781+0000","last_active":"2024-03-30T13:06:43.124590+0000","last_peered":"2024-03-30T13:06:43.124590+0000","last_clean":"2024-03-30T13:06:43.124590+0000","last_became_active":"2024-03-30T13:06:03.198523+0000","last_became_peered":"2024-03-30T13:06:03.198523+0000","last_unstale":"2024-03-30T13:06:43.124590+0000","last_undegraded":"2024-03-30T13:06:43.124590+0000","last_fullsized":"2024-03-30T13:06:43.124590+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:05:22.298091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:05:22.298091+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:06:50.590330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":666,"reported_epoch":294,"state":"active+clean","last_fresh":"2024-03-30T13:06:24.086460+0000","last_change":"2024-03-30T13:06:24.086526+0000","last_active":"2024-03-30T13:06:24.086460+0000","last_peered":"2024-03-30T13:06:24.086460+0000","last_clean":"2024-03-30T13:06:24.086460+0000","last_became_active":"2024-03-30T13:06:03.206122+0000","last_became_peered":"2024-03-30T13:06:03.206122+0000","last_unstale":"2024-03-30T13:06:24.086460+0000","last_undegraded":"2024-03-30T13:06:24.086460+0000","last_fullsized":"2024-03-30T13:06:24.086460+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:06:24.086281+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:06:24.086281+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:33.164463+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":20907644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":300858015,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106594304,"data_stored":106594304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":227,"seq":974957576224,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50292,"kb_used_data":18600,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321548,"statfs":{"total":96636764160,"available":96585265152,"internally_reserved":0,"allocated":19046400,"data_stored":17070088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":274,"seq":1176821039115,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56108,"kb_used_data":23972,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94315732,"statfs":{"total":96636764160,"available":96579309568,"internally_reserved":0,"allocated":24547328,"data_stored":22577775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6053,"internal_metadata":32893019},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824021,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47620,"kb_used_data":15036,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94324220,"statfs":{"total":96636764160,"available":96588001280,"internally_reserved":0,"allocated":15396864,"data_stored":13427311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509578,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47092,"kb_used_data":15912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324748,"statfs":{"total":96636764160,"available":96588541952,"internally_reserved":0,"allocated":16293888,"data_stored":14317576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285663,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43644,"kb_used_data":11956,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94328196,"statfs":{"total":96636764160,"available":96592072704,"internally_reserved":0,"allocated":12242944,"data_stored":10273391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509578,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48172,"kb_used_data":16992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323668,"statfs":{"total":96636764160,"available":96587436032,"internally_reserved":0,"allocated":17399808,"data_stored":15423496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574986,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47156,"kb_used_data":15980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324684,"statfs":{"total":96636764160,"available":96588476416,"internally_reserved":0,"allocated":16363520,"data_stored":14393967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8679424,"data_stored":8679424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12718080,"data_stored":12718080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20983808,"data_stored":20983808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:06:47.037 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:06:47.037 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:06:47.037 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2024-03-30T13:06:47.037 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-30T13:06:47.037 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-30T13:06:47.040 INFO:tasks.ceph.osd.7:Started 2024-03-30T13:06:47.040 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-30T13:06:47.110 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:06:47.108+0000 7fd5ec8707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:47.117 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:06:47.116+0000 7fd5ec8707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:47.118 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:06:47.116+0000 7fd5ec8707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:47.167 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T13:06:47.168 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T13:06:47.168 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T13:06:47.168 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T13:06:47.168 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T13:06:47.168 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T13:06:47.168 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-03-30T13:06:47.168 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-03-30T13:06:47.168 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-03-30T13:06:47.168 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-03-30T13:06:47.168 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-03-30T13:06:47.168 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T13:06:47.170 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:06:47.171 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-03-30T13:06:47.662 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:06:47.660+0000 7fd5ec8707c0 -1 Falling back to public interface 2024-03-30T13:06:51.500 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:06:51.496+0000 7fd5ec8707c0 -1 osd.7 292 log_to_monitors true 2024-03-30T13:06:52.153 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:06:52.152+0000 7fd5e063d640 -1 osd.7 292 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:06:52.172 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-30T13:06:52.303 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T13:06:52.304 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T13:06:52.304 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T13:06:52.304 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T13:06:52.317 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:06:52.317 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-03-30T13:06:52.317 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 7 2024-03-30T13:06:52.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:52.446+0000 7fee0a840640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:52.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:52.462+0000 7fee0a840640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:52.646 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:06:52.646+0000 7fbb38f08640 -1 mon.a@0(leader).osd e297 definitely_dead 0 2024-03-30T13:06:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.7. 2024-03-30T13:06:53.161 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-03-30T13:06:53.161 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:06:53.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:53.278+0000 7f376c70b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:53.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:53.294+0000 7f376c70b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:54.142 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:06:54.176 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:06:54.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:54.294+0000 7f36de353640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:54.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:54.310+0000 7f36de353640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:55.171 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:06:55.198 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:06:55.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:55.322+0000 7f9c7f7da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:55.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:55.338+0000 7f9c7f7da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:56.184 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:06:56.210 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:06:56.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:56.334+0000 7f78e1ee8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:56.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:56.354+0000 7f78e1ee8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:57.194 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:06:57.217 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:06:57.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:57.338+0000 7f99259e9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:57.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:57.354+0000 7f99259e9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:58.209 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:06:58.232 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:06:58.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:58.350+0000 7fbba6562640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:58.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:58.366+0000 7fbba6562640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:59.214 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:06:59.238 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:06:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:59.362+0000 7fa63c722640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:06:59.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:06:59.378+0000 7fa63c722640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:00.225 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:07:00.252 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:07:00.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:00.374+0000 7f68c7433640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:00.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:00.390+0000 7f68c7433640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:01.230 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:07:01.257 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:07:01.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:01.378+0000 7f4c5d952640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:01.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:01.394+0000 7f4c5d952640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:02.246 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:07:02.270 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:07:02.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:02.394+0000 7f00d1d7e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:02.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:02.414+0000 7f00d1d7e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:03.252 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:07:03.277 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:07:03.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:03.398+0000 7f1b63b88640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:03.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:03.418+0000 7f1b63b88640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:04.264 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:07:04.290 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:07:04.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:04.414+0000 7f753d296640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:04.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:04.430+0000 7f753d296640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:05.274 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:07:05.299 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:07:05.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:05.422+0000 7f06feaa4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:05.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:05.438+0000 7f06feaa4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:06.294 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:07:06.325 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:07:06.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:06.446+0000 7f3a811bf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:06.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:06.462+0000 7f3a811bf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:07:07.329 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:07:07.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:07.438+0000 7f8163b8c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:07.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:07.454+0000 7f8163b8c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:08.309 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:07:08.334 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:07:08.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:08.454+0000 7fbab0cd9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:08.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:08.470+0000 7fbab0cd9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:09.327 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:07:09.354 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:07:09.355 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:07:09.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:09.478+0000 7faef8e23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:09.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:09.494+0000 7faef8e23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:09.666 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:07:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:07:09.682 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":655,"stamp":"2024-03-30T13:07:09.325768+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":285967211,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":378592,"kb_used_data":123404,"kb_used_omap":37,"kb_used_meta":255002,"kb_avail":754596128,"statfs":{"total":773094113280,"available":772706435072,"internally_reserved":0,"allocated":126365696,"data_stored":109766890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38317,"internal_metadata":261122643},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.047863"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":692,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T13:07:04.271253+0000","last_change":"2024-03-30T13:06:34.434294+0000","last_active":"2024-03-30T13:07:04.271253+0000","last_peered":"2024-03-30T13:07:04.271253+0000","last_clean":"2024-03-30T13:07:04.271253+0000","last_became_active":"2024-03-30T13:06:03.198084+0000","last_became_peered":"2024-03-30T13:06:03.198084+0000","last_unstale":"2024-03-30T13:07:04.271253+0000","last_undegraded":"2024-03-30T13:07:04.271253+0000","last_fullsized":"2024-03-30T13:07:04.271253+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:06:34.434031+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:06:34.434031+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:37.346053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18352980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":650,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-03-30T13:07:05.280691+0000","last_change":"2024-03-30T13:06:54.155424+0000","last_active":"2024-03-30T13:07:05.280691+0000","last_peered":"2024-03-30T13:07:05.280691+0000","last_clean":"2024-03-30T13:07:05.280691+0000","last_became_active":"2024-03-30T13:06:54.153354+0000","last_became_peered":"2024-03-30T13:06:54.153354+0000","last_unstale":"2024-03-30T13:07:05.280691+0000","last_undegraded":"2024-03-30T13:07:05.280691+0000","last_fullsized":"2024-03-30T13:07:05.280691+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:06:14.049898+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:06:14.049898+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:24.394126+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":258,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.787138+0000","last_change":"2024-03-30T13:06:54.155534+0000","last_active":"2024-03-30T13:07:07.787138+0000","last_peered":"2024-03-30T13:07:07.787138+0000","last_clean":"2024-03-30T13:07:07.787138+0000","last_became_active":"2024-03-30T13:06:54.154160+0000","last_became_peered":"2024-03-30T13:06:54.154160+0000","last_unstale":"2024-03-30T13:07:07.787138+0000","last_undegraded":"2024-03-30T13:07:07.787138+0000","last_fullsized":"2024-03-30T13:07:07.787138+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:06:11.665099+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_clean_scrub_stamp":"2024-03-30T13:06:11.665099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:18.575523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":245,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.787065+0000","last_change":"2024-03-30T13:06:54.158081+0000","last_active":"2024-03-30T13:07:07.787065+0000","last_peered":"2024-03-30T13:07:07.787065+0000","last_clean":"2024-03-30T13:07:07.787065+0000","last_became_active":"2024-03-30T13:06:54.156927+0000","last_became_peered":"2024-03-30T13:06:54.156927+0000","last_unstale":"2024-03-30T13:07:07.787065+0000","last_undegraded":"2024-03-30T13:07:07.787065+0000","last_fullsized":"2024-03-30T13:07:07.787065+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:06:06.768191+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:06:06.768191+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:14.223931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":91,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.625557+0000","last_change":"2024-03-30T13:06:29.679546+0000","last_active":"2024-03-30T13:07:07.625557+0000","last_peered":"2024-03-30T13:07:07.625557+0000","last_clean":"2024-03-30T13:07:07.625557+0000","last_became_active":"2024-03-30T13:06:03.203196+0000","last_became_peered":"2024-03-30T13:06:03.203196+0000","last_unstale":"2024-03-30T13:07:07.625557+0000","last_undegraded":"2024-03-30T13:07:07.625557+0000","last_fullsized":"2024-03-30T13:07:07.625557+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:06:29.679251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:06:29.679251+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:46.794376+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":736,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.711643+0000","last_change":"2024-03-30T13:06:38.405609+0000","last_active":"2024-03-30T13:07:07.711643+0000","last_peered":"2024-03-30T13:07:07.711643+0000","last_clean":"2024-03-30T13:07:07.711643+0000","last_became_active":"2024-03-30T13:06:03.202467+0000","last_became_peered":"2024-03-30T13:06:03.202467+0000","last_unstale":"2024-03-30T13:07:07.711643+0000","last_undegraded":"2024-03-30T13:07:07.711643+0000","last_fullsized":"2024-03-30T13:07:07.711643+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:06:38.405328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:06:38.405328+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:04.811288+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":45579924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":272,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.787055+0000","last_change":"2024-03-30T13:06:54.388448+0000","last_active":"2024-03-30T13:07:07.787055+0000","last_peered":"2024-03-30T13:07:07.787055+0000","last_clean":"2024-03-30T13:07:07.787055+0000","last_became_active":"2024-03-30T13:06:54.155351+0000","last_became_peered":"2024-03-30T13:06:54.155351+0000","last_unstale":"2024-03-30T13:07:07.787055+0000","last_undegraded":"2024-03-30T13:07:07.787055+0000","last_fullsized":"2024-03-30T13:07:07.787055+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:05:54.746578+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:05:54.746578+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:07.946810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":739,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.786365+0000","last_change":"2024-03-30T13:06:29.144702+0000","last_active":"2024-03-30T13:07:07.786365+0000","last_peered":"2024-03-30T13:07:07.786365+0000","last_clean":"2024-03-30T13:07:07.786365+0000","last_became_active":"2024-03-30T13:06:03.206177+0000","last_became_peered":"2024-03-30T13:06:03.206177+0000","last_unstale":"2024-03-30T13:07:07.786365+0000","last_undegraded":"2024-03-30T13:07:07.786365+0000","last_fullsized":"2024-03-30T13:07:07.786365+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:06:29.144448+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:06:29.144448+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:37.166703+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":820,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.711857+0000","last_change":"2024-03-30T13:07:05.492098+0000","last_active":"2024-03-30T13:07:07.711857+0000","last_peered":"2024-03-30T13:07:07.711857+0000","last_clean":"2024-03-30T13:07:07.711857+0000","last_became_active":"2024-03-30T13:06:54.150549+0000","last_became_peered":"2024-03-30T13:06:54.150549+0000","last_unstale":"2024-03-30T13:07:07.711857+0000","last_undegraded":"2024-03-30T13:07:07.711857+0000","last_fullsized":"2024-03-30T13:07:07.711857+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:05.491728+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:40.339270+0000","last_clean_scrub_stamp":"2024-03-30T13:07:05.491728+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:34.307985+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":560,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-03-30T13:07:05.280464+0000","last_change":"2024-03-30T13:06:34.924067+0000","last_active":"2024-03-30T13:07:05.280464+0000","last_peered":"2024-03-30T13:07:05.280464+0000","last_clean":"2024-03-30T13:07:05.280464+0000","last_became_active":"2024-03-30T13:05:07.445793+0000","last_became_peered":"2024-03-30T13:05:07.445793+0000","last_unstale":"2024-03-30T13:07:05.280464+0000","last_undegraded":"2024-03-30T13:07:05.280464+0000","last_fullsized":"2024-03-30T13:07:05.280464+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:06:34.923827+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:06:34.923827+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:01.504665+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":714,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T13:07:04.270932+0000","last_change":"2024-03-30T13:06:19.487398+0000","last_active":"2024-03-30T13:07:04.270932+0000","last_peered":"2024-03-30T13:07:04.270932+0000","last_clean":"2024-03-30T13:07:04.270932+0000","last_became_active":"2024-03-30T13:05:07.445475+0000","last_became_peered":"2024-03-30T13:05:07.445475+0000","last_unstale":"2024-03-30T13:07:04.270932+0000","last_undegraded":"2024-03-30T13:07:04.270932+0000","last_fullsized":"2024-03-30T13:07:04.270932+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:06:19.487011+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:06:19.487011+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:46.181120+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":734,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T13:07:04.271024+0000","last_change":"2024-03-30T13:06:05.397483+0000","last_active":"2024-03-30T13:07:04.271024+0000","last_peered":"2024-03-30T13:07:04.271024+0000","last_clean":"2024-03-30T13:07:04.271024+0000","last_became_active":"2024-03-30T13:06:03.206037+0000","last_became_peered":"2024-03-30T13:06:03.206037+0000","last_unstale":"2024-03-30T13:07:04.271024+0000","last_undegraded":"2024-03-30T13:07:04.271024+0000","last_fullsized":"2024-03-30T13:07:04.271024+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:06:05.397104+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:06:05.397104+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:28.298853+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":97,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.625544+0000","last_change":"2024-03-30T13:06:43.656954+0000","last_active":"2024-03-30T13:07:07.625544+0000","last_peered":"2024-03-30T13:07:07.625544+0000","last_clean":"2024-03-30T13:07:07.625544+0000","last_became_active":"2024-03-30T13:06:03.194120+0000","last_became_peered":"2024-03-30T13:06:03.194120+0000","last_unstale":"2024-03-30T13:07:07.625544+0000","last_undegraded":"2024-03-30T13:07:07.625544+0000","last_fullsized":"2024-03-30T13:07:07.625544+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:06:43.656581+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_clean_scrub_stamp":"2024-03-30T13:06:43.656581+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:56.020823+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":218,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.787070+0000","last_change":"2024-03-30T13:07:01.439676+0000","last_active":"2024-03-30T13:07:07.787070+0000","last_peered":"2024-03-30T13:07:07.787070+0000","last_clean":"2024-03-30T13:07:07.787070+0000","last_became_active":"2024-03-30T13:06:54.153682+0000","last_became_peered":"2024-03-30T13:06:54.153682+0000","last_unstale":"2024-03-30T13:07:07.787070+0000","last_undegraded":"2024-03-30T13:07:07.787070+0000","last_fullsized":"2024-03-30T13:07:07.787070+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:07:01.439296+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:07:01.439296+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:21.420094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":97,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.625359+0000","last_change":"2024-03-30T13:06:54.715146+0000","last_active":"2024-03-30T13:07:07.625359+0000","last_peered":"2024-03-30T13:07:07.625359+0000","last_clean":"2024-03-30T13:07:07.625359+0000","last_became_active":"2024-03-30T13:06:03.198523+0000","last_became_peered":"2024-03-30T13:06:03.198523+0000","last_unstale":"2024-03-30T13:07:07.625359+0000","last_undegraded":"2024-03-30T13:07:07.625359+0000","last_fullsized":"2024-03-30T13:07:07.625359+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:06:54.714864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:06:54.714864+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:08.724715+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":701,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.786731+0000","last_change":"2024-03-30T13:06:24.086526+0000","last_active":"2024-03-30T13:07:07.786731+0000","last_peered":"2024-03-30T13:07:07.786731+0000","last_clean":"2024-03-30T13:07:07.786731+0000","last_became_active":"2024-03-30T13:06:03.206122+0000","last_became_peered":"2024-03-30T13:06:03.206122+0000","last_unstale":"2024-03-30T13:07:07.786731+0000","last_undegraded":"2024-03-30T13:07:07.786731+0000","last_fullsized":"2024-03-30T13:07:07.786731+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:06:24.086281+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:06:24.086281+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:33.164463+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":20907644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":285967211,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":227,"seq":974957576228,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50468,"kb_used_data":18768,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321372,"statfs":{"total":96636764160,"available":96585084928,"internally_reserved":0,"allocated":19218432,"data_stored":17148657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":274,"seq":1176821039120,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55680,"kb_used_data":23536,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94316160,"statfs":{"total":96636764160,"available":96579747840,"internally_reserved":0,"allocated":24100864,"data_stored":22017537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6053,"internal_metadata":32893019},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824026,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47824,"kb_used_data":15228,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94324016,"statfs":{"total":96636764160,"available":96587792384,"internally_reserved":0,"allocated":15593472,"data_stored":13516946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509582,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41172,"kb_used_data":9984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330668,"statfs":{"total":96636764160,"available":96594604032,"internally_reserved":0,"allocated":10223616,"data_stored":8153841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285668,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40552,"kb_used_data":8852,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94331288,"statfs":{"total":96636764160,"available":96595238912,"internally_reserved":0,"allocated":9064448,"data_stored":6981121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509582,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48320,"kb_used_data":17136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323520,"statfs":{"total":96636764160,"available":96587284480,"internally_reserved":0,"allocated":17547264,"data_stored":15491041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":298,"seq":1279900254213,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47208,"kb_used_data":13716,"kb_used_omap":10,"kb_used_meta":33461,"kb_avail":94324632,"statfs":{"total":96636764160,"available":96588423168,"internally_reserved":0,"allocated":14045184,"data_stored":11968658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10986,"internal_metadata":34264342},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574991,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47368,"kb_used_data":16184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324472,"statfs":{"total":96636764160,"available":96588259328,"internally_reserved":0,"allocated":16572416,"data_stored":14489089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:07:09.684 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:07:09.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:09.794+0000 7ff1f580f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:09.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:09.806+0000 7ff1f580f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:09.970 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:07:09.971 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:07:09.987 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":655,"stamp":"2024-03-30T13:07:09.325768+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":285967211,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":378592,"kb_used_data":123404,"kb_used_omap":37,"kb_used_meta":255002,"kb_avail":754596128,"statfs":{"total":773094113280,"available":772706435072,"internally_reserved":0,"allocated":126365696,"data_stored":109766890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38317,"internal_metadata":261122643},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.047863"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":692,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T13:07:04.271253+0000","last_change":"2024-03-30T13:06:34.434294+0000","last_active":"2024-03-30T13:07:04.271253+0000","last_peered":"2024-03-30T13:07:04.271253+0000","last_clean":"2024-03-30T13:07:04.271253+0000","last_became_active":"2024-03-30T13:06:03.198084+0000","last_became_peered":"2024-03-30T13:06:03.198084+0000","last_unstale":"2024-03-30T13:07:04.271253+0000","last_undegraded":"2024-03-30T13:07:04.271253+0000","last_fullsized":"2024-03-30T13:07:04.271253+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:06:34.434031+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:06:34.434031+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:37.346053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18352980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":650,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-03-30T13:07:05.280691+0000","last_change":"2024-03-30T13:06:54.155424+0000","last_active":"2024-03-30T13:07:05.280691+0000","last_peered":"2024-03-30T13:07:05.280691+0000","last_clean":"2024-03-30T13:07:05.280691+0000","last_became_active":"2024-03-30T13:06:54.153354+0000","last_became_peered":"2024-03-30T13:06:54.153354+0000","last_unstale":"2024-03-30T13:07:05.280691+0000","last_undegraded":"2024-03-30T13:07:05.280691+0000","last_fullsized":"2024-03-30T13:07:05.280691+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:06:14.049898+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:06:14.049898+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:24.394126+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":258,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.787138+0000","last_change":"2024-03-30T13:06:54.155534+0000","last_active":"2024-03-30T13:07:07.787138+0000","last_peered":"2024-03-30T13:07:07.787138+0000","last_clean":"2024-03-30T13:07:07.787138+0000","last_became_active":"2024-03-30T13:06:54.154160+0000","last_became_peered":"2024-03-30T13:06:54.154160+0000","last_unstale":"2024-03-30T13:07:07.787138+0000","last_undegraded":"2024-03-30T13:07:07.787138+0000","last_fullsized":"2024-03-30T13:07:07.787138+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:06:11.665099+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_clean_scrub_stamp":"2024-03-30T13:06:11.665099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:18.575523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":245,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.787065+0000","last_change":"2024-03-30T13:06:54.158081+0000","last_active":"2024-03-30T13:07:07.787065+0000","last_peered":"2024-03-30T13:07:07.787065+0000","last_clean":"2024-03-30T13:07:07.787065+0000","last_became_active":"2024-03-30T13:06:54.156927+0000","last_became_peered":"2024-03-30T13:06:54.156927+0000","last_unstale":"2024-03-30T13:07:07.787065+0000","last_undegraded":"2024-03-30T13:07:07.787065+0000","last_fullsized":"2024-03-30T13:07:07.787065+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:06:06.768191+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:06:06.768191+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:14.223931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":91,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.625557+0000","last_change":"2024-03-30T13:06:29.679546+0000","last_active":"2024-03-30T13:07:07.625557+0000","last_peered":"2024-03-30T13:07:07.625557+0000","last_clean":"2024-03-30T13:07:07.625557+0000","last_became_active":"2024-03-30T13:06:03.203196+0000","last_became_peered":"2024-03-30T13:06:03.203196+0000","last_unstale":"2024-03-30T13:07:07.625557+0000","last_undegraded":"2024-03-30T13:07:07.625557+0000","last_fullsized":"2024-03-30T13:07:07.625557+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:06:29.679251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:06:29.679251+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:46.794376+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":736,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.711643+0000","last_change":"2024-03-30T13:06:38.405609+0000","last_active":"2024-03-30T13:07:07.711643+0000","last_peered":"2024-03-30T13:07:07.711643+0000","last_clean":"2024-03-30T13:07:07.711643+0000","last_became_active":"2024-03-30T13:06:03.202467+0000","last_became_peered":"2024-03-30T13:06:03.202467+0000","last_unstale":"2024-03-30T13:07:07.711643+0000","last_undegraded":"2024-03-30T13:07:07.711643+0000","last_fullsized":"2024-03-30T13:07:07.711643+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:06:38.405328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:06:38.405328+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:04.811288+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":45579924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":272,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.787055+0000","last_change":"2024-03-30T13:06:54.388448+0000","last_active":"2024-03-30T13:07:07.787055+0000","last_peered":"2024-03-30T13:07:07.787055+0000","last_clean":"2024-03-30T13:07:07.787055+0000","last_became_active":"2024-03-30T13:06:54.155351+0000","last_became_peered":"2024-03-30T13:06:54.155351+0000","last_unstale":"2024-03-30T13:07:07.787055+0000","last_undegraded":"2024-03-30T13:07:07.787055+0000","last_fullsized":"2024-03-30T13:07:07.787055+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:05:54.746578+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:05:54.746578+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:07.946810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":739,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.786365+0000","last_change":"2024-03-30T13:06:29.144702+0000","last_active":"2024-03-30T13:07:07.786365+0000","last_peered":"2024-03-30T13:07:07.786365+0000","last_clean":"2024-03-30T13:07:07.786365+0000","last_became_active":"2024-03-30T13:06:03.206177+0000","last_became_peered":"2024-03-30T13:06:03.206177+0000","last_unstale":"2024-03-30T13:07:07.786365+0000","last_undegraded":"2024-03-30T13:07:07.786365+0000","last_fullsized":"2024-03-30T13:07:07.786365+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:06:29.144448+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:06:29.144448+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:37.166703+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":820,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.711857+0000","last_change":"2024-03-30T13:07:05.492098+0000","last_active":"2024-03-30T13:07:07.711857+0000","last_peered":"2024-03-30T13:07:07.711857+0000","last_clean":"2024-03-30T13:07:07.711857+0000","last_became_active":"2024-03-30T13:06:54.150549+0000","last_became_peered":"2024-03-30T13:06:54.150549+0000","last_unstale":"2024-03-30T13:07:07.711857+0000","last_undegraded":"2024-03-30T13:07:07.711857+0000","last_fullsized":"2024-03-30T13:07:07.711857+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:05.491728+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:40.339270+0000","last_clean_scrub_stamp":"2024-03-30T13:07:05.491728+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:34.307985+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":560,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-03-30T13:07:05.280464+0000","last_change":"2024-03-30T13:06:34.924067+0000","last_active":"2024-03-30T13:07:05.280464+0000","last_peered":"2024-03-30T13:07:05.280464+0000","last_clean":"2024-03-30T13:07:05.280464+0000","last_became_active":"2024-03-30T13:05:07.445793+0000","last_became_peered":"2024-03-30T13:05:07.445793+0000","last_unstale":"2024-03-30T13:07:05.280464+0000","last_undegraded":"2024-03-30T13:07:05.280464+0000","last_fullsized":"2024-03-30T13:07:05.280464+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:06:34.923827+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:06:34.923827+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:01.504665+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":714,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T13:07:04.270932+0000","last_change":"2024-03-30T13:06:19.487398+0000","last_active":"2024-03-30T13:07:04.270932+0000","last_peered":"2024-03-30T13:07:04.270932+0000","last_clean":"2024-03-30T13:07:04.270932+0000","last_became_active":"2024-03-30T13:05:07.445475+0000","last_became_peered":"2024-03-30T13:05:07.445475+0000","last_unstale":"2024-03-30T13:07:04.270932+0000","last_undegraded":"2024-03-30T13:07:04.270932+0000","last_fullsized":"2024-03-30T13:07:04.270932+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:06:19.487011+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:06:19.487011+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:46.181120+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":734,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T13:07:04.271024+0000","last_change":"2024-03-30T13:06:05.397483+0000","last_active":"2024-03-30T13:07:04.271024+0000","last_peered":"2024-03-30T13:07:04.271024+0000","last_clean":"2024-03-30T13:07:04.271024+0000","last_became_active":"2024-03-30T13:06:03.206037+0000","last_became_peered":"2024-03-30T13:06:03.206037+0000","last_unstale":"2024-03-30T13:07:04.271024+0000","last_undegraded":"2024-03-30T13:07:04.271024+0000","last_fullsized":"2024-03-30T13:07:04.271024+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:06:05.397104+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:06:05.397104+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:28.298853+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":97,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.625544+0000","last_change":"2024-03-30T13:06:43.656954+0000","last_active":"2024-03-30T13:07:07.625544+0000","last_peered":"2024-03-30T13:07:07.625544+0000","last_clean":"2024-03-30T13:07:07.625544+0000","last_became_active":"2024-03-30T13:06:03.194120+0000","last_became_peered":"2024-03-30T13:06:03.194120+0000","last_unstale":"2024-03-30T13:07:07.625544+0000","last_undegraded":"2024-03-30T13:07:07.625544+0000","last_fullsized":"2024-03-30T13:07:07.625544+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:06:43.656581+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_clean_scrub_stamp":"2024-03-30T13:06:43.656581+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:56.020823+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":218,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.787070+0000","last_change":"2024-03-30T13:07:01.439676+0000","last_active":"2024-03-30T13:07:07.787070+0000","last_peered":"2024-03-30T13:07:07.787070+0000","last_clean":"2024-03-30T13:07:07.787070+0000","last_became_active":"2024-03-30T13:06:54.153682+0000","last_became_peered":"2024-03-30T13:06:54.153682+0000","last_unstale":"2024-03-30T13:07:07.787070+0000","last_undegraded":"2024-03-30T13:07:07.787070+0000","last_fullsized":"2024-03-30T13:07:07.787070+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:07:01.439296+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:07:01.439296+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:21.420094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":97,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.625359+0000","last_change":"2024-03-30T13:06:54.715146+0000","last_active":"2024-03-30T13:07:07.625359+0000","last_peered":"2024-03-30T13:07:07.625359+0000","last_clean":"2024-03-30T13:07:07.625359+0000","last_became_active":"2024-03-30T13:06:03.198523+0000","last_became_peered":"2024-03-30T13:06:03.198523+0000","last_unstale":"2024-03-30T13:07:07.625359+0000","last_undegraded":"2024-03-30T13:07:07.625359+0000","last_fullsized":"2024-03-30T13:07:07.625359+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:06:54.714864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:06:54.714864+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:08.724715+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":701,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.786731+0000","last_change":"2024-03-30T13:06:24.086526+0000","last_active":"2024-03-30T13:07:07.786731+0000","last_peered":"2024-03-30T13:07:07.786731+0000","last_clean":"2024-03-30T13:07:07.786731+0000","last_became_active":"2024-03-30T13:06:03.206122+0000","last_became_peered":"2024-03-30T13:06:03.206122+0000","last_unstale":"2024-03-30T13:07:07.786731+0000","last_undegraded":"2024-03-30T13:07:07.786731+0000","last_fullsized":"2024-03-30T13:07:07.786731+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:06:24.086281+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:06:24.086281+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:33.164463+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":20907644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":285967211,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":227,"seq":974957576228,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50468,"kb_used_data":18768,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321372,"statfs":{"total":96636764160,"available":96585084928,"internally_reserved":0,"allocated":19218432,"data_stored":17148657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":274,"seq":1176821039120,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55680,"kb_used_data":23536,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94316160,"statfs":{"total":96636764160,"available":96579747840,"internally_reserved":0,"allocated":24100864,"data_stored":22017537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6053,"internal_metadata":32893019},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824026,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47824,"kb_used_data":15228,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94324016,"statfs":{"total":96636764160,"available":96587792384,"internally_reserved":0,"allocated":15593472,"data_stored":13516946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509582,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41172,"kb_used_data":9984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330668,"statfs":{"total":96636764160,"available":96594604032,"internally_reserved":0,"allocated":10223616,"data_stored":8153841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285668,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40552,"kb_used_data":8852,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94331288,"statfs":{"total":96636764160,"available":96595238912,"internally_reserved":0,"allocated":9064448,"data_stored":6981121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509582,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48320,"kb_used_data":17136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323520,"statfs":{"total":96636764160,"available":96587284480,"internally_reserved":0,"allocated":17547264,"data_stored":15491041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":298,"seq":1279900254213,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47208,"kb_used_data":13716,"kb_used_omap":10,"kb_used_meta":33461,"kb_avail":94324632,"statfs":{"total":96636764160,"available":96588423168,"internally_reserved":0,"allocated":14045184,"data_stored":11968658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10986,"internal_metadata":34264342},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574991,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47368,"kb_used_data":16184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324472,"statfs":{"total":96636764160,"available":96588259328,"internally_reserved":0,"allocated":16572416,"data_stored":14489089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:07:09.989 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:07:09.989 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:07:10.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:10.098+0000 7f658f5a9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:10.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:10.110+0000 7f658f5a9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:10.283 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:07:10.283 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":314,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:07:09.320980+0000","last_up_change":"2024-03-30T13:06:53.124864+0000","last_in_change":"2024-03-30T13:06:53.124864+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":77,"up_from":83,"up_thru":293,"down_at":78,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6818","nonce":3104224893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6821","nonce":3104224893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6828","nonce":3104224893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6825","nonce":3104224893}]},"public_addr":"172.21.15.67:6818/3104224893","cluster_addr":"172.21.15.67:6821/3104224893","heartbeat_back_addr":"172.21.15.67:6828/3104224893","heartbeat_front_addr":"172.21.15.67:6825/3104224893","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":293,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":298,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6801","nonce":1504348724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6803","nonce":1504348724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6807","nonce":1504348724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6805","nonce":1504348724}]},"public_addr":"172.21.15.67:6801/1504348724","cluster_addr":"172.21.15.67:6803/1504348724","heartbeat_back_addr":"172.21.15.67:6807/1504348724","heartbeat_front_addr":"172.21.15.67:6805/1504348724","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":135,"last_clean_end":243,"up_from":250,"up_thru":298,"down_at":244,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6803","nonce":3986522908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6807","nonce":3986522908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6815","nonce":3986522908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6811","nonce":3986522908}]},"public_addr":"172.21.15.82:6803/3986522908","cluster_addr":"172.21.15.82:6807/3986522908","heartbeat_back_addr":"172.21.15.82:6815/3986522908","heartbeat_front_addr":"172.21.15.82:6811/3986522908","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":156,"last_clean_end":266,"up_from":274,"up_thru":293,"down_at":267,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":2286739047},{"type":"v1","addr":"172.21.15.82:6817","nonce":2286739047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":2286739047},{"type":"v1","addr":"172.21.15.82:6819","nonce":2286739047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":2286739047},{"type":"v1","addr":"172.21.15.82:6823","nonce":2286739047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":2286739047},{"type":"v1","addr":"172.21.15.82:6821","nonce":2286739047}]},"public_addr":"172.21.15.82:6817/2286739047","cluster_addr":"172.21.15.82:6819/2286739047","heartbeat_back_addr":"172.21.15.82:6823/2286739047","heartbeat_front_addr":"172.21.15.82:6821/2286739047","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":222,"up_from":227,"up_thru":276,"down_at":223,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3068584906},{"type":"v1","addr":"172.21.15.82:6801","nonce":3068584906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3068584906},{"type":"v1","addr":"172.21.15.82:6806","nonce":3068584906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3068584906},{"type":"v1","addr":"172.21.15.82:6814","nonce":3068584906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3068584906},{"type":"v1","addr":"172.21.15.82:6810","nonce":3068584906}]},"public_addr":"172.21.15.82:6801/3068584906","cluster_addr":"172.21.15.82:6806/3068584906","heartbeat_back_addr":"172.21.15.82:6814/3068584906","heartbeat_front_addr":"172.21.15.82:6810/3068584906","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":206,"last_clean_end":292,"up_from":298,"up_thru":298,"down_at":293,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3894469331},{"type":"v1","addr":"172.21.15.82:6825","nonce":3894469331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3894469331},{"type":"v1","addr":"172.21.15.82:6827","nonce":3894469331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3894469331},{"type":"v1","addr":"172.21.15.82:6831","nonce":3894469331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3894469331},{"type":"v1","addr":"172.21.15.82:6829","nonce":3894469331}]},"public_addr":"172.21.15.82:6825/3894469331","cluster_addr":"172.21.15.82:6827/3894469331","heartbeat_back_addr":"172.21.15.82:6831/3894469331","heartbeat_front_addr":"172.21.15.82:6829/3894469331","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:07:04.796993+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:07:06.436617+0000","dead_epoch":78},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:06:42.718293+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:06:09.409172+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:06:57.279609+0000","dead_epoch":244},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:07:03.994444+0000","dead_epoch":267},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:06:44.638787+0000","dead_epoch":223},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:06:52.464189+0000","dead_epoch":293}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:07:10.302 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:07:10.302 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:07:10.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:10.426+0000 7f6067f18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:10.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:10.446+0000 7f6067f18640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:10.613 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:07:10.633 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:07:10.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:10.746+0000 7f6813289640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:10.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:10.762+0000 7f6813289640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:10.938 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:07:10.938 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:07:10.957 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:07:10.957 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:07:10.957 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:07:10.957 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:07:10.957 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 3, 1, 6, 4, 5, 7] 2024-03-30T13:07:10.957 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:07:10.958 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:07:11.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:11.066+0000 7f7a8b929640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:11.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:11.078+0000 7f7a8b929640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:11.243 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:07:11.243 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:07:11.260 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":656,"stamp":"2024-03-30T13:07:09.981765+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":285967211,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":378696,"kb_used_data":123512,"kb_used_omap":37,"kb_used_meta":255002,"kb_avail":754596024,"statfs":{"total":773094113280,"available":772706328576,"internally_reserved":0,"allocated":126476288,"data_stored":109816523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38317,"internal_metadata":261122643},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001051"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":692,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T13:07:04.271253+0000","last_change":"2024-03-30T13:06:34.434294+0000","last_active":"2024-03-30T13:07:04.271253+0000","last_peered":"2024-03-30T13:07:04.271253+0000","last_clean":"2024-03-30T13:07:04.271253+0000","last_became_active":"2024-03-30T13:06:03.198084+0000","last_became_peered":"2024-03-30T13:06:03.198084+0000","last_unstale":"2024-03-30T13:07:04.271253+0000","last_undegraded":"2024-03-30T13:07:04.271253+0000","last_fullsized":"2024-03-30T13:07:04.271253+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:06:34.434031+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:06:34.434031+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:37.346053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18352980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":650,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-03-30T13:07:05.280691+0000","last_change":"2024-03-30T13:06:54.155424+0000","last_active":"2024-03-30T13:07:05.280691+0000","last_peered":"2024-03-30T13:07:05.280691+0000","last_clean":"2024-03-30T13:07:05.280691+0000","last_became_active":"2024-03-30T13:06:54.153354+0000","last_became_peered":"2024-03-30T13:06:54.153354+0000","last_unstale":"2024-03-30T13:07:05.280691+0000","last_undegraded":"2024-03-30T13:07:05.280691+0000","last_fullsized":"2024-03-30T13:07:05.280691+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:06:14.049898+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:06:14.049898+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:24.394126+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":258,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.787138+0000","last_change":"2024-03-30T13:06:54.155534+0000","last_active":"2024-03-30T13:07:07.787138+0000","last_peered":"2024-03-30T13:07:07.787138+0000","last_clean":"2024-03-30T13:07:07.787138+0000","last_became_active":"2024-03-30T13:06:54.154160+0000","last_became_peered":"2024-03-30T13:06:54.154160+0000","last_unstale":"2024-03-30T13:07:07.787138+0000","last_undegraded":"2024-03-30T13:07:07.787138+0000","last_fullsized":"2024-03-30T13:07:07.787138+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:06:11.665099+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_clean_scrub_stamp":"2024-03-30T13:06:11.665099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:18.575523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":245,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.787065+0000","last_change":"2024-03-30T13:06:54.158081+0000","last_active":"2024-03-30T13:07:07.787065+0000","last_peered":"2024-03-30T13:07:07.787065+0000","last_clean":"2024-03-30T13:07:07.787065+0000","last_became_active":"2024-03-30T13:06:54.156927+0000","last_became_peered":"2024-03-30T13:06:54.156927+0000","last_unstale":"2024-03-30T13:07:07.787065+0000","last_undegraded":"2024-03-30T13:07:07.787065+0000","last_fullsized":"2024-03-30T13:07:07.787065+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:06:06.768191+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:06:06.768191+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:14.223931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":91,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.625557+0000","last_change":"2024-03-30T13:06:29.679546+0000","last_active":"2024-03-30T13:07:07.625557+0000","last_peered":"2024-03-30T13:07:07.625557+0000","last_clean":"2024-03-30T13:07:07.625557+0000","last_became_active":"2024-03-30T13:06:03.203196+0000","last_became_peered":"2024-03-30T13:06:03.203196+0000","last_unstale":"2024-03-30T13:07:07.625557+0000","last_undegraded":"2024-03-30T13:07:07.625557+0000","last_fullsized":"2024-03-30T13:07:07.625557+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:06:29.679251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:06:29.679251+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:46.794376+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":736,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.711643+0000","last_change":"2024-03-30T13:06:38.405609+0000","last_active":"2024-03-30T13:07:07.711643+0000","last_peered":"2024-03-30T13:07:07.711643+0000","last_clean":"2024-03-30T13:07:07.711643+0000","last_became_active":"2024-03-30T13:06:03.202467+0000","last_became_peered":"2024-03-30T13:06:03.202467+0000","last_unstale":"2024-03-30T13:07:07.711643+0000","last_undegraded":"2024-03-30T13:07:07.711643+0000","last_fullsized":"2024-03-30T13:07:07.711643+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:06:38.405328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:06:38.405328+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:04.811288+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":45579924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":272,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.787055+0000","last_change":"2024-03-30T13:06:54.388448+0000","last_active":"2024-03-30T13:07:07.787055+0000","last_peered":"2024-03-30T13:07:07.787055+0000","last_clean":"2024-03-30T13:07:07.787055+0000","last_became_active":"2024-03-30T13:06:54.155351+0000","last_became_peered":"2024-03-30T13:06:54.155351+0000","last_unstale":"2024-03-30T13:07:07.787055+0000","last_undegraded":"2024-03-30T13:07:07.787055+0000","last_fullsized":"2024-03-30T13:07:07.787055+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:05:54.746578+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:05:54.746578+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:07.946810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":739,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.786365+0000","last_change":"2024-03-30T13:06:29.144702+0000","last_active":"2024-03-30T13:07:07.786365+0000","last_peered":"2024-03-30T13:07:07.786365+0000","last_clean":"2024-03-30T13:07:07.786365+0000","last_became_active":"2024-03-30T13:06:03.206177+0000","last_became_peered":"2024-03-30T13:06:03.206177+0000","last_unstale":"2024-03-30T13:07:07.786365+0000","last_undegraded":"2024-03-30T13:07:07.786365+0000","last_fullsized":"2024-03-30T13:07:07.786365+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:06:29.144448+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:06:29.144448+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:37.166703+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":820,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.711857+0000","last_change":"2024-03-30T13:07:05.492098+0000","last_active":"2024-03-30T13:07:07.711857+0000","last_peered":"2024-03-30T13:07:07.711857+0000","last_clean":"2024-03-30T13:07:07.711857+0000","last_became_active":"2024-03-30T13:06:54.150549+0000","last_became_peered":"2024-03-30T13:06:54.150549+0000","last_unstale":"2024-03-30T13:07:07.711857+0000","last_undegraded":"2024-03-30T13:07:07.711857+0000","last_fullsized":"2024-03-30T13:07:07.711857+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:05.491728+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:40.339270+0000","last_clean_scrub_stamp":"2024-03-30T13:07:05.491728+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:34.307985+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":560,"reported_epoch":310,"state":"active+clean","last_fresh":"2024-03-30T13:07:05.280464+0000","last_change":"2024-03-30T13:06:34.924067+0000","last_active":"2024-03-30T13:07:05.280464+0000","last_peered":"2024-03-30T13:07:05.280464+0000","last_clean":"2024-03-30T13:07:05.280464+0000","last_became_active":"2024-03-30T13:05:07.445793+0000","last_became_peered":"2024-03-30T13:05:07.445793+0000","last_unstale":"2024-03-30T13:07:05.280464+0000","last_undegraded":"2024-03-30T13:07:05.280464+0000","last_fullsized":"2024-03-30T13:07:05.280464+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:06:34.923827+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:06:34.923827+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:01.504665+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":714,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T13:07:04.270932+0000","last_change":"2024-03-30T13:06:19.487398+0000","last_active":"2024-03-30T13:07:04.270932+0000","last_peered":"2024-03-30T13:07:04.270932+0000","last_clean":"2024-03-30T13:07:04.270932+0000","last_became_active":"2024-03-30T13:05:07.445475+0000","last_became_peered":"2024-03-30T13:05:07.445475+0000","last_unstale":"2024-03-30T13:07:04.270932+0000","last_undegraded":"2024-03-30T13:07:04.270932+0000","last_fullsized":"2024-03-30T13:07:04.270932+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:06:19.487011+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:06:19.487011+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:46.181120+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":734,"reported_epoch":309,"state":"active+clean","last_fresh":"2024-03-30T13:07:04.271024+0000","last_change":"2024-03-30T13:06:05.397483+0000","last_active":"2024-03-30T13:07:04.271024+0000","last_peered":"2024-03-30T13:07:04.271024+0000","last_clean":"2024-03-30T13:07:04.271024+0000","last_became_active":"2024-03-30T13:06:03.206037+0000","last_became_peered":"2024-03-30T13:06:03.206037+0000","last_unstale":"2024-03-30T13:07:04.271024+0000","last_undegraded":"2024-03-30T13:07:04.271024+0000","last_fullsized":"2024-03-30T13:07:04.271024+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:06:05.397104+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:06:05.397104+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:28.298853+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":97,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.625544+0000","last_change":"2024-03-30T13:06:43.656954+0000","last_active":"2024-03-30T13:07:07.625544+0000","last_peered":"2024-03-30T13:07:07.625544+0000","last_clean":"2024-03-30T13:07:07.625544+0000","last_became_active":"2024-03-30T13:06:03.194120+0000","last_became_peered":"2024-03-30T13:06:03.194120+0000","last_unstale":"2024-03-30T13:07:07.625544+0000","last_undegraded":"2024-03-30T13:07:07.625544+0000","last_fullsized":"2024-03-30T13:07:07.625544+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:06:43.656581+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_clean_scrub_stamp":"2024-03-30T13:06:43.656581+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:56.020823+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":218,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.787070+0000","last_change":"2024-03-30T13:07:01.439676+0000","last_active":"2024-03-30T13:07:07.787070+0000","last_peered":"2024-03-30T13:07:07.787070+0000","last_clean":"2024-03-30T13:07:07.787070+0000","last_became_active":"2024-03-30T13:06:54.153682+0000","last_became_peered":"2024-03-30T13:06:54.153682+0000","last_unstale":"2024-03-30T13:07:07.787070+0000","last_undegraded":"2024-03-30T13:07:07.787070+0000","last_fullsized":"2024-03-30T13:07:07.787070+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:07:01.439296+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:07:01.439296+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:21.420094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":97,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.625359+0000","last_change":"2024-03-30T13:06:54.715146+0000","last_active":"2024-03-30T13:07:07.625359+0000","last_peered":"2024-03-30T13:07:07.625359+0000","last_clean":"2024-03-30T13:07:07.625359+0000","last_became_active":"2024-03-30T13:06:03.198523+0000","last_became_peered":"2024-03-30T13:06:03.198523+0000","last_unstale":"2024-03-30T13:07:07.625359+0000","last_undegraded":"2024-03-30T13:07:07.625359+0000","last_fullsized":"2024-03-30T13:07:07.625359+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:06:54.714864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:06:54.714864+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:08.724715+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":701,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.786731+0000","last_change":"2024-03-30T13:06:24.086526+0000","last_active":"2024-03-30T13:07:07.786731+0000","last_peered":"2024-03-30T13:07:07.786731+0000","last_clean":"2024-03-30T13:07:07.786731+0000","last_became_active":"2024-03-30T13:06:03.206122+0000","last_became_peered":"2024-03-30T13:06:03.206122+0000","last_unstale":"2024-03-30T13:07:07.786731+0000","last_undegraded":"2024-03-30T13:07:07.786731+0000","last_fullsized":"2024-03-30T13:07:07.786731+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:06:24.086281+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:06:24.086281+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:33.164463+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":20907644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":285967211,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":227,"seq":974957576228,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50468,"kb_used_data":18768,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321372,"statfs":{"total":96636764160,"available":96585084928,"internally_reserved":0,"allocated":19218432,"data_stored":17148657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":274,"seq":1176821039120,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55680,"kb_used_data":23536,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94316160,"statfs":{"total":96636764160,"available":96579747840,"internally_reserved":0,"allocated":24100864,"data_stored":22017537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6053,"internal_metadata":32893019},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824026,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47824,"kb_used_data":15228,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94324016,"statfs":{"total":96636764160,"available":96587792384,"internally_reserved":0,"allocated":15593472,"data_stored":13516946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509583,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41212,"kb_used_data":10032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330628,"statfs":{"total":96636764160,"available":96594563072,"internally_reserved":0,"allocated":10272768,"data_stored":8175889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285668,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40552,"kb_used_data":8852,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94331288,"statfs":{"total":96636764160,"available":96595238912,"internally_reserved":0,"allocated":9064448,"data_stored":6981121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509583,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48384,"kb_used_data":17196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323456,"statfs":{"total":96636764160,"available":96587218944,"internally_reserved":0,"allocated":17608704,"data_stored":15518626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":298,"seq":1279900254213,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47208,"kb_used_data":13716,"kb_used_omap":10,"kb_used_meta":33461,"kb_avail":94324632,"statfs":{"total":96636764160,"available":96588423168,"internally_reserved":0,"allocated":14045184,"data_stored":11968658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10986,"internal_metadata":34264342},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574991,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47368,"kb_used_data":16184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324472,"statfs":{"total":96636764160,"available":96588259328,"internally_reserved":0,"allocated":16572416,"data_stored":14489089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:07:11.261 INFO:tasks.thrashosds.thrasher:Choosing PG 1.1 with acting set [5, 2, 0] 2024-03-30T13:07:11.261 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 2, 3, 1, 6, 4, 5, 7] 2024-03-30T13:07:11.261 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-30T13:07:11.261 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:07:11.321 INFO:tasks.ceph.osd.5:Stopped 2024-03-30T13:07:11.321 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 2, 3, 1, 6, 4, 5, 7] 2024-03-30T13:07:11.321 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 5 2024-03-30T13:07:11.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:11.442+0000 7f0d811b4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:11.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:11.458+0000 7f0d811b4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:11.639 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:07:11.642+0000 7fbb38f08640 -1 mon.a@0(leader).osd e315 definitely_dead 0 2024-03-30T13:07:12.352 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.5. 2024-03-30T13:07:12.382 INFO:tasks.thrashosds.thrasher:dead_osds=[5], live_osds=[0, 2, 3, 1, 6, 4, 7] 2024-03-30T13:07:12.382 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:07:12.382 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:07:12.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:12.501+0000 7f3903d9d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:12.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:12.513+0000 7f3903d9d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:12.688 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:07:12.689 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:07:12.708 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":659,"stamp":"2024-03-30T13:07:12.356098+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":285967211,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":323100,"kb_used_data":100072,"kb_used_omap":31,"kb_used_meta":222880,"kb_avail":660279780,"statfs":{"total":676457349120,"available":676126494720,"internally_reserved":0,"allocated":102473728,"data_stored":87843032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32264,"internal_metadata":228229624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.374871"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":699,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-30T13:07:08.317430+0000","last_change":"2024-03-30T13:06:34.434294+0000","last_active":"2024-03-30T13:07:08.317430+0000","last_peered":"2024-03-30T13:07:08.317430+0000","last_clean":"2024-03-30T13:07:08.317430+0000","last_became_active":"2024-03-30T13:06:03.198084+0000","last_became_peered":"2024-03-30T13:06:03.198084+0000","last_unstale":"2024-03-30T13:07:08.317430+0000","last_undegraded":"2024-03-30T13:07:08.317430+0000","last_fullsized":"2024-03-30T13:07:08.317430+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:06:34.434031+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:06:34.434031+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:37.346053+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18352980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":659,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-30T13:07:11.345061+0000","last_change":"2024-03-30T13:06:54.155424+0000","last_active":"2024-03-30T13:07:11.345061+0000","last_peered":"2024-03-30T13:07:11.345061+0000","last_clean":"2024-03-30T13:07:11.345061+0000","last_became_active":"2024-03-30T13:06:54.153354+0000","last_became_peered":"2024-03-30T13:06:54.153354+0000","last_unstale":"2024-03-30T13:07:11.345061+0000","last_undegraded":"2024-03-30T13:07:11.345061+0000","last_fullsized":"2024-03-30T13:07:11.345061+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:06:14.049898+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:06:14.049898+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:24.394126+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":258,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.787138+0000","last_change":"2024-03-30T13:06:54.155534+0000","last_active":"2024-03-30T13:07:07.787138+0000","last_peered":"2024-03-30T13:07:07.787138+0000","last_clean":"2024-03-30T13:07:07.787138+0000","last_became_active":"2024-03-30T13:06:54.154160+0000","last_became_peered":"2024-03-30T13:06:54.154160+0000","last_unstale":"2024-03-30T13:07:07.787138+0000","last_undegraded":"2024-03-30T13:07:07.787138+0000","last_fullsized":"2024-03-30T13:07:07.787138+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:06:11.665099+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_clean_scrub_stamp":"2024-03-30T13:06:11.665099+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:18.575523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":245,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.787065+0000","last_change":"2024-03-30T13:06:54.158081+0000","last_active":"2024-03-30T13:07:07.787065+0000","last_peered":"2024-03-30T13:07:07.787065+0000","last_clean":"2024-03-30T13:07:07.787065+0000","last_became_active":"2024-03-30T13:06:54.156927+0000","last_became_peered":"2024-03-30T13:06:54.156927+0000","last_unstale":"2024-03-30T13:07:07.787065+0000","last_undegraded":"2024-03-30T13:07:07.787065+0000","last_fullsized":"2024-03-30T13:07:07.787065+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:06:06.768191+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:06:06.768191+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:14.223931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":91,"reported_epoch":312,"state":"stale+active+clean","last_fresh":"2024-03-30T13:07:07.625557+0000","last_change":"2024-03-30T13:06:29.679546+0000","last_active":"2024-03-30T13:07:07.625557+0000","last_peered":"2024-03-30T13:07:07.625557+0000","last_clean":"2024-03-30T13:07:07.625557+0000","last_became_active":"2024-03-30T13:06:03.203196+0000","last_became_peered":"2024-03-30T13:06:03.203196+0000","last_unstale":"2024-03-30T13:07:11.343368+0000","last_undegraded":"2024-03-30T13:07:07.625557+0000","last_fullsized":"2024-03-30T13:07:07.625557+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:06:29.679251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:06:29.679251+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:46.794376+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":736,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.711643+0000","last_change":"2024-03-30T13:06:38.405609+0000","last_active":"2024-03-30T13:07:07.711643+0000","last_peered":"2024-03-30T13:07:07.711643+0000","last_clean":"2024-03-30T13:07:07.711643+0000","last_became_active":"2024-03-30T13:06:03.202467+0000","last_became_peered":"2024-03-30T13:06:03.202467+0000","last_unstale":"2024-03-30T13:07:07.711643+0000","last_undegraded":"2024-03-30T13:07:07.711643+0000","last_fullsized":"2024-03-30T13:07:07.711643+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:06:38.405328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:06:38.405328+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:04.811288+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":45579924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":272,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.787055+0000","last_change":"2024-03-30T13:06:54.388448+0000","last_active":"2024-03-30T13:07:07.787055+0000","last_peered":"2024-03-30T13:07:07.787055+0000","last_clean":"2024-03-30T13:07:07.787055+0000","last_became_active":"2024-03-30T13:06:54.155351+0000","last_became_peered":"2024-03-30T13:06:54.155351+0000","last_unstale":"2024-03-30T13:07:07.787055+0000","last_undegraded":"2024-03-30T13:07:07.787055+0000","last_fullsized":"2024-03-30T13:07:07.787055+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:05:54.746578+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:05:54.746578+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:07.946810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":739,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.786365+0000","last_change":"2024-03-30T13:06:29.144702+0000","last_active":"2024-03-30T13:07:07.786365+0000","last_peered":"2024-03-30T13:07:07.786365+0000","last_clean":"2024-03-30T13:07:07.786365+0000","last_became_active":"2024-03-30T13:06:03.206177+0000","last_became_peered":"2024-03-30T13:06:03.206177+0000","last_unstale":"2024-03-30T13:07:07.786365+0000","last_undegraded":"2024-03-30T13:07:07.786365+0000","last_fullsized":"2024-03-30T13:07:07.786365+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:06:29.144448+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:06:29.144448+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:37.166703+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":820,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.711857+0000","last_change":"2024-03-30T13:07:05.492098+0000","last_active":"2024-03-30T13:07:07.711857+0000","last_peered":"2024-03-30T13:07:07.711857+0000","last_clean":"2024-03-30T13:07:07.711857+0000","last_became_active":"2024-03-30T13:06:54.150549+0000","last_became_peered":"2024-03-30T13:06:54.150549+0000","last_unstale":"2024-03-30T13:07:07.711857+0000","last_undegraded":"2024-03-30T13:07:07.711857+0000","last_fullsized":"2024-03-30T13:07:07.711857+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:05.491728+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:40.339270+0000","last_clean_scrub_stamp":"2024-03-30T13:07:05.491728+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:34.307985+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":570,"reported_epoch":315,"state":"active+clean","last_fresh":"2024-03-30T13:07:11.344948+0000","last_change":"2024-03-30T13:06:34.924067+0000","last_active":"2024-03-30T13:07:11.344948+0000","last_peered":"2024-03-30T13:07:11.344948+0000","last_clean":"2024-03-30T13:07:11.344948+0000","last_became_active":"2024-03-30T13:05:07.445793+0000","last_became_peered":"2024-03-30T13:05:07.445793+0000","last_unstale":"2024-03-30T13:07:11.344948+0000","last_undegraded":"2024-03-30T13:07:11.344948+0000","last_fullsized":"2024-03-30T13:07:11.344948+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:06:34.923827+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:06:34.923827+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:01.504665+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":721,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-30T13:07:08.317294+0000","last_change":"2024-03-30T13:06:19.487398+0000","last_active":"2024-03-30T13:07:08.317294+0000","last_peered":"2024-03-30T13:07:08.317294+0000","last_clean":"2024-03-30T13:07:08.317294+0000","last_became_active":"2024-03-30T13:05:07.445475+0000","last_became_peered":"2024-03-30T13:05:07.445475+0000","last_unstale":"2024-03-30T13:07:08.317294+0000","last_undegraded":"2024-03-30T13:07:08.317294+0000","last_fullsized":"2024-03-30T13:07:08.317294+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:06:19.487011+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:06:19.487011+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:46.181120+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":741,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-03-30T13:07:08.317315+0000","last_change":"2024-03-30T13:06:05.397483+0000","last_active":"2024-03-30T13:07:08.317315+0000","last_peered":"2024-03-30T13:07:08.317315+0000","last_clean":"2024-03-30T13:07:08.317315+0000","last_became_active":"2024-03-30T13:06:03.206037+0000","last_became_peered":"2024-03-30T13:06:03.206037+0000","last_unstale":"2024-03-30T13:07:08.317315+0000","last_undegraded":"2024-03-30T13:07:08.317315+0000","last_fullsized":"2024-03-30T13:07:08.317315+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:06:05.397104+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:06:05.397104+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:28.298853+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":97,"reported_epoch":312,"state":"stale+active+clean","last_fresh":"2024-03-30T13:07:07.625544+0000","last_change":"2024-03-30T13:06:43.656954+0000","last_active":"2024-03-30T13:07:07.625544+0000","last_peered":"2024-03-30T13:07:07.625544+0000","last_clean":"2024-03-30T13:07:07.625544+0000","last_became_active":"2024-03-30T13:06:03.194120+0000","last_became_peered":"2024-03-30T13:06:03.194120+0000","last_unstale":"2024-03-30T13:07:11.343372+0000","last_undegraded":"2024-03-30T13:07:07.625544+0000","last_fullsized":"2024-03-30T13:07:07.625544+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:06:43.656581+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_clean_scrub_stamp":"2024-03-30T13:06:43.656581+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:56.020823+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":218,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.787070+0000","last_change":"2024-03-30T13:07:01.439676+0000","last_active":"2024-03-30T13:07:07.787070+0000","last_peered":"2024-03-30T13:07:07.787070+0000","last_clean":"2024-03-30T13:07:07.787070+0000","last_became_active":"2024-03-30T13:06:54.153682+0000","last_became_peered":"2024-03-30T13:06:54.153682+0000","last_unstale":"2024-03-30T13:07:07.787070+0000","last_undegraded":"2024-03-30T13:07:07.787070+0000","last_fullsized":"2024-03-30T13:07:07.787070+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:07:01.439296+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:07:01.439296+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:21.420094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":97,"reported_epoch":312,"state":"stale+active+clean","last_fresh":"2024-03-30T13:07:07.625359+0000","last_change":"2024-03-30T13:06:54.715146+0000","last_active":"2024-03-30T13:07:07.625359+0000","last_peered":"2024-03-30T13:07:07.625359+0000","last_clean":"2024-03-30T13:07:07.625359+0000","last_became_active":"2024-03-30T13:06:03.198523+0000","last_became_peered":"2024-03-30T13:06:03.198523+0000","last_unstale":"2024-03-30T13:07:11.343374+0000","last_undegraded":"2024-03-30T13:07:07.625359+0000","last_fullsized":"2024-03-30T13:07:07.625359+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:06:54.714864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:06:54.714864+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:08.724715+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":701,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-30T13:07:07.786731+0000","last_change":"2024-03-30T13:06:24.086526+0000","last_active":"2024-03-30T13:07:07.786731+0000","last_peered":"2024-03-30T13:07:07.786731+0000","last_clean":"2024-03-30T13:07:07.786731+0000","last_became_active":"2024-03-30T13:06:03.206122+0000","last_became_peered":"2024-03-30T13:06:03.206122+0000","last_unstale":"2024-03-30T13:07:07.786731+0000","last_undegraded":"2024-03-30T13:07:07.786731+0000","last_fullsized":"2024-03-30T13:07:07.786731+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:06:24.086281+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:06:24.086281+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:33.164463+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":20907644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":285967211,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":227,"seq":974957576230,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50508,"kb_used_data":18816,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321332,"statfs":{"total":96636764160,"available":96585043968,"internally_reserved":0,"allocated":19267584,"data_stored":17170705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824027,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47868,"kb_used_data":15276,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94323972,"statfs":{"total":96636764160,"available":96587747328,"internally_reserved":0,"allocated":15642624,"data_stored":13538944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509583,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41212,"kb_used_data":10032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330628,"statfs":{"total":96636764160,"available":96594563072,"internally_reserved":0,"allocated":10272768,"data_stored":8175889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285668,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40552,"kb_used_data":8852,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94331288,"statfs":{"total":96636764160,"available":96595238912,"internally_reserved":0,"allocated":9064448,"data_stored":6981121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509583,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48384,"kb_used_data":17196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323456,"statfs":{"total":96636764160,"available":96587218944,"internally_reserved":0,"allocated":17608704,"data_stored":15518626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":298,"seq":1279900254213,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47208,"kb_used_data":13716,"kb_used_omap":10,"kb_used_meta":33461,"kb_avail":94324632,"statfs":{"total":96636764160,"available":96588423168,"internally_reserved":0,"allocated":14045184,"data_stored":11968658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10986,"internal_metadata":34264342},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574991,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47368,"kb_used_data":16184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324472,"statfs":{"total":96636764160,"available":96588259328,"internally_reserved":0,"allocated":16572416,"data_stored":14489089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:07:12.709 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:07:37.710 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:07:37.711 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:07:37.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:37.841+0000 7f7c7289b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:37.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:37.857+0000 7f7c7289b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:38.036 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:07:38.037 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:07:38.054 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":674,"stamp":"2024-03-30T13:07:37.989728+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":338841307,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":343404,"kb_used_data":120428,"kb_used_omap":31,"kb_used_meta":222880,"kb_avail":660259476,"statfs":{"total":676457349120,"available":676105703424,"internally_reserved":0,"allocated":123318272,"data_stored":108404793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32264,"internal_metadata":228229624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003337"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":720,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-03-30T13:07:15.661663+0000","last_change":"2024-03-30T13:07:13.583411+0000","last_active":"2024-03-30T13:07:15.661663+0000","last_peered":"2024-03-30T13:07:15.661663+0000","last_clean":"2024-03-30T13:07:15.661663+0000","last_became_active":"2024-03-30T13:07:13.398181+0000","last_became_peered":"2024-03-30T13:07:13.398181+0000","last_unstale":"2024-03-30T13:07:15.661663+0000","last_undegraded":"2024-03-30T13:07:15.661663+0000","last_fullsized":"2024-03-30T13:07:15.661663+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:06:34.434031+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:06:34.434031+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:37.907707+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22941225,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":674,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-03-30T13:07:26.626065+0000","last_change":"2024-03-30T13:07:26.626194+0000","last_active":"2024-03-30T13:07:26.626065+0000","last_peered":"2024-03-30T13:07:26.626065+0000","last_clean":"2024-03-30T13:07:26.626065+0000","last_became_active":"2024-03-30T13:06:54.153354+0000","last_became_peered":"2024-03-30T13:06:54.153354+0000","last_unstale":"2024-03-30T13:07:26.626065+0000","last_undegraded":"2024-03-30T13:07:26.626065+0000","last_fullsized":"2024-03-30T13:07:26.626065+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:07:26.625854+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:07:26.625854+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:30.881671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":278,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-03-30T13:07:23.683930+0000","last_change":"2024-03-30T13:07:23.684003+0000","last_active":"2024-03-30T13:07:23.683930+0000","last_peered":"2024-03-30T13:07:23.683930+0000","last_clean":"2024-03-30T13:07:23.683930+0000","last_became_active":"2024-03-30T13:06:54.154160+0000","last_became_peered":"2024-03-30T13:06:54.154160+0000","last_unstale":"2024-03-30T13:07:23.683930+0000","last_undegraded":"2024-03-30T13:07:23.683930+0000","last_fullsized":"2024-03-30T13:07:23.683930+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:07:23.683745+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_clean_scrub_stamp":"2024-03-30T13:07:23.683745+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:49.948436+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":265,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-03-30T13:07:16.522919+0000","last_change":"2024-03-30T13:07:16.522990+0000","last_active":"2024-03-30T13:07:16.522919+0000","last_peered":"2024-03-30T13:07:16.522919+0000","last_clean":"2024-03-30T13:07:16.522919+0000","last_became_active":"2024-03-30T13:06:54.156927+0000","last_became_peered":"2024-03-30T13:06:54.156927+0000","last_unstale":"2024-03-30T13:07:16.522919+0000","last_undegraded":"2024-03-30T13:07:16.522919+0000","last_fullsized":"2024-03-30T13:07:16.522919+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:16.522745+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:07:16.522745+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:40.030402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":17,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-03-30T13:07:15.557485+0000","last_change":"2024-03-30T13:07:14.274480+0000","last_active":"2024-03-30T13:07:15.557485+0000","last_peered":"2024-03-30T13:07:15.557485+0000","last_clean":"2024-03-30T13:07:15.557485+0000","last_became_active":"2024-03-30T13:07:13.389862+0000","last_became_peered":"2024-03-30T13:07:13.389862+0000","last_unstale":"2024-03-30T13:07:15.557485+0000","last_undegraded":"2024-03-30T13:07:15.557485+0000","last_fullsized":"2024-03-30T13:07:15.557485+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:06:29.679251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:06:29.679251+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:32.560165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":764,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-03-30T13:07:15.188228+0000","last_change":"2024-03-30T13:07:14.258244+0000","last_active":"2024-03-30T13:07:15.188228+0000","last_peered":"2024-03-30T13:07:15.188228+0000","last_clean":"2024-03-30T13:07:15.188228+0000","last_became_active":"2024-03-30T13:07:13.397604+0000","last_became_peered":"2024-03-30T13:07:13.397604+0000","last_unstale":"2024-03-30T13:07:15.188228+0000","last_undegraded":"2024-03-30T13:07:15.188228+0000","last_fullsized":"2024-03-30T13:07:15.188228+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:06:38.405328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:06:38.405328+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:39.772367+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":56974905,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":292,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-03-30T13:07:14.397182+0000","last_change":"2024-03-30T13:07:08.403925+0000","last_active":"2024-03-30T13:07:14.397182+0000","last_peered":"2024-03-30T13:07:14.397182+0000","last_clean":"2024-03-30T13:07:14.397182+0000","last_became_active":"2024-03-30T13:06:54.155351+0000","last_became_peered":"2024-03-30T13:06:54.155351+0000","last_unstale":"2024-03-30T13:07:14.397182+0000","last_undegraded":"2024-03-30T13:07:14.397182+0000","last_fullsized":"2024-03-30T13:07:14.397182+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:07:08.403663+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:07:08.403663+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:24.852917+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":765,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-03-30T13:07:15.107590+0000","last_change":"2024-03-30T13:07:14.228993+0000","last_active":"2024-03-30T13:07:15.107590+0000","last_peered":"2024-03-30T13:07:15.107590+0000","last_clean":"2024-03-30T13:07:15.107590+0000","last_became_active":"2024-03-30T13:07:13.396206+0000","last_became_peered":"2024-03-30T13:07:13.396206+0000","last_unstale":"2024-03-30T13:07:15.107590+0000","last_undegraded":"2024-03-30T13:07:15.107590+0000","last_fullsized":"2024-03-30T13:07:15.107590+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:06:29.144448+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:06:29.144448+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:40.248014+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":51411920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":842,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-03-30T13:07:15.188383+0000","last_change":"2024-03-30T13:07:13.451407+0000","last_active":"2024-03-30T13:07:15.188383+0000","last_peered":"2024-03-30T13:07:15.188383+0000","last_clean":"2024-03-30T13:07:15.188383+0000","last_became_active":"2024-03-30T13:07:13.400642+0000","last_became_peered":"2024-03-30T13:07:13.400642+0000","last_unstale":"2024-03-30T13:07:15.188383+0000","last_undegraded":"2024-03-30T13:07:15.188383+0000","last_fullsized":"2024-03-30T13:07:15.188383+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:05.491728+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:40.339270+0000","last_clean_scrub_stamp":"2024-03-30T13:07:05.491728+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:27.419941+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":65983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":576,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-03-30T13:07:15.557392+0000","last_change":"2024-03-30T13:06:34.924067+0000","last_active":"2024-03-30T13:07:15.557392+0000","last_peered":"2024-03-30T13:07:15.557392+0000","last_clean":"2024-03-30T13:07:15.557392+0000","last_became_active":"2024-03-30T13:05:07.445793+0000","last_became_peered":"2024-03-30T13:05:07.445793+0000","last_unstale":"2024-03-30T13:07:15.557392+0000","last_undegraded":"2024-03-30T13:07:15.557392+0000","last_fullsized":"2024-03-30T13:07:15.557392+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:06:34.923827+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:06:34.923827+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:01.504665+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":731,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-03-30T13:07:15.661265+0000","last_change":"2024-03-30T13:06:19.487398+0000","last_active":"2024-03-30T13:07:15.661265+0000","last_peered":"2024-03-30T13:07:15.661265+0000","last_clean":"2024-03-30T13:07:15.661265+0000","last_became_active":"2024-03-30T13:05:07.445475+0000","last_became_peered":"2024-03-30T13:05:07.445475+0000","last_unstale":"2024-03-30T13:07:15.661265+0000","last_undegraded":"2024-03-30T13:07:15.661265+0000","last_fullsized":"2024-03-30T13:07:15.661265+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:06:19.487011+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:02:18.933526+0000","last_clean_scrub_stamp":"2024-03-30T13:06:19.487011+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:46.181120+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":782,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-03-30T13:07:24.693516+0000","last_change":"2024-03-30T13:07:24.693658+0000","last_active":"2024-03-30T13:07:24.693516+0000","last_peered":"2024-03-30T13:07:24.693516+0000","last_clean":"2024-03-30T13:07:24.693516+0000","last_became_active":"2024-03-30T13:07:13.399935+0000","last_became_peered":"2024-03-30T13:07:13.399935+0000","last_unstale":"2024-03-30T13:07:24.693516+0000","last_undegraded":"2024-03-30T13:07:24.693516+0000","last_fullsized":"2024-03-30T13:07:24.693516+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:07:24.693213+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:07:24.693213+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:51.099595+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":57731253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":23,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-03-30T13:07:15.661281+0000","last_change":"2024-03-30T13:07:13.544808+0000","last_active":"2024-03-30T13:07:15.661281+0000","last_peered":"2024-03-30T13:07:15.661281+0000","last_clean":"2024-03-30T13:07:15.661281+0000","last_became_active":"2024-03-30T13:07:13.394855+0000","last_became_peered":"2024-03-30T13:07:13.394855+0000","last_unstale":"2024-03-30T13:07:15.661281+0000","last_undegraded":"2024-03-30T13:07:15.661281+0000","last_fullsized":"2024-03-30T13:07:15.661281+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:06:43.656581+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:04:21.312610+0000","last_clean_scrub_stamp":"2024-03-30T13:06:43.656581+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:48.704597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":230,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-03-30T13:07:14.397182+0000","last_change":"2024-03-30T13:07:01.439676+0000","last_active":"2024-03-30T13:07:14.397182+0000","last_peered":"2024-03-30T13:07:14.397182+0000","last_clean":"2024-03-30T13:07:14.397182+0000","last_became_active":"2024-03-30T13:06:54.153682+0000","last_became_peered":"2024-03-30T13:06:54.153682+0000","last_unstale":"2024-03-30T13:07:14.397182+0000","last_undegraded":"2024-03-30T13:07:14.397182+0000","last_fullsized":"2024-03-30T13:07:14.397182+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:07:01.439296+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:07:01.439296+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:21.420094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":23,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-03-30T13:07:15.557502+0000","last_change":"2024-03-30T13:07:13.471156+0000","last_active":"2024-03-30T13:07:15.557502+0000","last_peered":"2024-03-30T13:07:15.557502+0000","last_clean":"2024-03-30T13:07:15.557502+0000","last_became_active":"2024-03-30T13:07:13.389674+0000","last_became_peered":"2024-03-30T13:07:13.389674+0000","last_unstale":"2024-03-30T13:07:15.557502+0000","last_undegraded":"2024-03-30T13:07:15.557502+0000","last_fullsized":"2024-03-30T13:07:15.557502+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:06:54.714864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:06:54.714864+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:00.276463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":724,"reported_epoch":318,"state":"active+clean","last_fresh":"2024-03-30T13:07:15.107463+0000","last_change":"2024-03-30T13:07:14.294073+0000","last_active":"2024-03-30T13:07:15.107463+0000","last_peered":"2024-03-30T13:07:15.107463+0000","last_clean":"2024-03-30T13:07:15.107463+0000","last_became_active":"2024-03-30T13:07:13.402424+0000","last_became_peered":"2024-03-30T13:07:13.402424+0000","last_unstale":"2024-03-30T13:07:15.107463+0000","last_undegraded":"2024-03-30T13:07:15.107463+0000","last_fullsized":"2024-03-30T13:07:15.107463+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:06:24.086281+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:06:24.086281+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:07:38.087881+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":26134555,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140,"num_bytes_recovered":338841307,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":116637696,"data_stored":116637696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":227,"seq":974957576235,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56128,"kb_used_data":24440,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94315712,"statfs":{"total":96636764160,"available":96579289088,"internally_reserved":0,"allocated":25026560,"data_stored":22896063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824032,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52404,"kb_used_data":19820,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94319436,"statfs":{"total":96636764160,"available":96583102464,"internally_reserved":0,"allocated":20295680,"data_stored":18165183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509588,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43900,"kb_used_data":12720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327940,"statfs":{"total":96636764160,"available":96591810560,"internally_reserved":0,"allocated":13025280,"data_stored":10894783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285673,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45836,"kb_used_data":14148,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94326004,"statfs":{"total":96636764160,"available":96589828096,"internally_reserved":0,"allocated":14487552,"data_stored":12357055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509588,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48448,"kb_used_data":17268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323392,"statfs":{"total":96636764160,"available":96587153408,"internally_reserved":0,"allocated":17682432,"data_stored":15551935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":298,"seq":1279900254218,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47292,"kb_used_data":13812,"kb_used_omap":10,"kb_used_meta":33461,"kb_avail":94324548,"statfs":{"total":96636764160,"available":96588337152,"internally_reserved":0,"allocated":14143488,"data_stored":12012991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10986,"internal_metadata":34264342},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574997,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49396,"kb_used_data":18220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322444,"statfs":{"total":96636764160,"available":96586182656,"internally_reserved":0,"allocated":18657280,"data_stored":16526783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":10629120,"data_stored":10629120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9166848,"data_stored":9166848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":14798848,"data_stored":14798848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16437248,"data_stored":16437248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":21168128,"data_stored":21168128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:07:38.056 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:07:38.056 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:07:38.056 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-03-30T13:07:38.056 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-30T13:07:38.056 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-30T13:07:38.059 INFO:tasks.ceph.osd.5:Started 2024-03-30T13:07:38.060 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-30T13:07:38.128 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:07:38.128+0000 7ff20a3a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:38.138 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:07:38.136+0000 7ff20a3a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:38.138 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:07:38.136+0000 7ff20a3a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:38.177 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T13:07:38.177 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T13:07:38.177 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T13:07:38.177 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T13:07:38.177 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T13:07:38.177 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T13:07:38.177 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-03-30T13:07:38.178 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-03-30T13:07:38.178 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-03-30T13:07:38.178 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-03-30T13:07:38.178 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-03-30T13:07:38.178 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T13:07:38.180 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:07:38.180 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-03-30T13:07:38.678 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:07:38.676+0000 7ff20a3a07c0 -1 Falling back to public interface 2024-03-30T13:07:42.458 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:07:42.456+0000 7ff20a3a07c0 -1 osd.5 314 log_to_monitors true 2024-03-30T13:07:43.157 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:07:43.156+0000 7ff1fe16d640 -1 osd.5 314 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:07:43.181 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-30T13:07:43.308 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T13:07:43.309 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T13:07:43.309 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T13:07:43.309 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T13:07:43.322 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:07:43.322 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-03-30T13:07:43.323 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 5 2024-03-30T13:07:43.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:43.449+0000 7f1769497640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:43.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:43.465+0000 7f1769497640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:43.646 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:07:43.645+0000 7fbb38f08640 -1 mon.a@0(leader).osd e319 definitely_dead 0 2024-03-30T13:07:44.122 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.5. 2024-03-30T13:07:44.157 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-03-30T13:07:44.157 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:07:44.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:44.277+0000 7f07097bf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:44.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:44.493+0000 7f0702ffd640 0 --2- 172.21.15.67:0/358630130 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f0704072930 0x7f0704141c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-03-30T13:07:44.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:44.497+0000 7f07097bf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:45.147 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:07:45.180 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:07:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:45.297+0000 7f63ae8c7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:45.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:45.313+0000 7f63ae8c7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:46.164 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:07:46.186 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:07:46.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:46.309+0000 7f20f3077640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:46.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:46.325+0000 7f20f3077640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:47.166 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:07:47.192 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:07:47.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:47.313+0000 7fca6e983640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:47.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:47.329+0000 7fca6e983640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:48.177 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:07:48.202 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:07:48.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:48.321+0000 7fae1ae0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:48.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:48.337+0000 7fae1ae0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:49.196 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:07:49.221 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:07:49.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:49.337+0000 7f9cfa456640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:49.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:49.353+0000 7f9cfa456640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:50.207 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:07:50.234 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:07:50.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:50.357+0000 7f2b0e2bc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:50.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:50.377+0000 7f2b0e2bc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:51.210 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:07:51.235 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:07:51.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:51.357+0000 7f3e2f4e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:51.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:51.373+0000 7f3e2f4e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:52.225 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:07:52.252 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:07:52.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:52.373+0000 7f6fb2b2f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:52.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:52.389+0000 7f6fb2b2f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:53.230 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:07:53.257 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:07:53.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:53.389+0000 7fbf455bf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:53.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:53.405+0000 7fbf455bf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:54.245 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:07:54.269 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:07:54.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:54.389+0000 7fbcb53d7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:54.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:54.405+0000 7fbcb53d7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:55.248 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:07:55.276 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:07:55.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:55.397+0000 7fe46b051640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:55.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:55.413+0000 7fe46b051640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:56.274 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:07:56.301 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:07:56.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:56.421+0000 7f5ef5f04640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:56.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:56.437+0000 7f5ef5f04640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:57.267 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:07:57.293 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:07:57.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:57.409+0000 7f3633a4e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:57.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:57.429+0000 7f3633a4e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:58.284 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:07:58.311 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:07:58.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:58.429+0000 7f82b2cb0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:58.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:58.445+0000 7f82b2cb0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:59.293 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:07:59.319 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:07:59.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:59.437+0000 7f910b503640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:07:59.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:07:59.457+0000 7f910b503640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:00.295 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:08:00.323 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:08:00.323 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:08:00.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:00.437+0000 7f981944f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:00.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:00.449+0000 7f981944f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:00.618 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:08:00.619 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:08:00.634 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":703,"stamp":"2024-03-30T13:08:00.299408+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":318370678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":381188,"kb_used_data":125564,"kb_used_omap":39,"kb_used_meta":255448,"kb_avail":754593532,"statfs":{"total":773094113280,"available":772703776768,"internally_reserved":0,"allocated":128577536,"data_stored":110771436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39968,"internal_metadata":261579744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.030276"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":764,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.290859+0000","last_change":"2024-03-30T13:07:45.158080+0000","last_active":"2024-03-30T13:07:58.290859+0000","last_peered":"2024-03-30T13:07:58.290859+0000","last_clean":"2024-03-30T13:07:58.290859+0000","last_became_active":"2024-03-30T13:07:45.156785+0000","last_became_peered":"2024-03-30T13:07:45.156785+0000","last_unstale":"2024-03-30T13:07:58.290859+0000","last_undegraded":"2024-03-30T13:07:58.290859+0000","last_fullsized":"2024-03-30T13:07:58.290859+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:38.717555+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:07:38.717555+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:57.207615+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22941225,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":701,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-30T13:07:56.271016+0000","last_change":"2024-03-30T13:07:26.626194+0000","last_active":"2024-03-30T13:07:56.271016+0000","last_peered":"2024-03-30T13:07:56.271016+0000","last_clean":"2024-03-30T13:07:56.271016+0000","last_became_active":"2024-03-30T13:06:54.153354+0000","last_became_peered":"2024-03-30T13:06:54.153354+0000","last_unstale":"2024-03-30T13:07:56.271016+0000","last_undegraded":"2024-03-30T13:07:56.271016+0000","last_fullsized":"2024-03-30T13:07:56.271016+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:07:26.625854+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:07:26.625854+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:30.881671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":305,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.117258+0000","last_change":"2024-03-30T13:07:23.684003+0000","last_active":"2024-03-30T13:07:58.117258+0000","last_peered":"2024-03-30T13:07:58.117258+0000","last_clean":"2024-03-30T13:07:58.117258+0000","last_became_active":"2024-03-30T13:06:54.154160+0000","last_became_peered":"2024-03-30T13:06:54.154160+0000","last_unstale":"2024-03-30T13:07:58.117258+0000","last_undegraded":"2024-03-30T13:07:58.117258+0000","last_fullsized":"2024-03-30T13:07:58.117258+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:07:23.683745+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_clean_scrub_stamp":"2024-03-30T13:07:23.683745+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:49.948436+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":292,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.117155+0000","last_change":"2024-03-30T13:07:16.522990+0000","last_active":"2024-03-30T13:07:58.117155+0000","last_peered":"2024-03-30T13:07:58.117155+0000","last_clean":"2024-03-30T13:07:58.117155+0000","last_became_active":"2024-03-30T13:06:54.156927+0000","last_became_peered":"2024-03-30T13:06:54.156927+0000","last_unstale":"2024-03-30T13:07:58.117155+0000","last_undegraded":"2024-03-30T13:07:58.117155+0000","last_fullsized":"2024-03-30T13:07:58.117155+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:16.522745+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:07:16.522745+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:40.030402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":92,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.765104+0000","last_change":"2024-03-30T13:07:45.158822+0000","last_active":"2024-03-30T13:07:58.765104+0000","last_peered":"2024-03-30T13:07:58.765104+0000","last_clean":"2024-03-30T13:07:58.765104+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:07:58.765104+0000","last_undegraded":"2024-03-30T13:07:58.765104+0000","last_fullsized":"2024-03-30T13:07:58.765104+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:07:37.576416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:07:37.576416+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:50.550338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":801,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-30T13:07:55.720689+0000","last_change":"2024-03-30T13:07:45.157963+0000","last_active":"2024-03-30T13:07:55.720689+0000","last_peered":"2024-03-30T13:07:55.720689+0000","last_clean":"2024-03-30T13:07:55.720689+0000","last_became_active":"2024-03-30T13:07:45.156174+0000","last_became_peered":"2024-03-30T13:07:45.156174+0000","last_unstale":"2024-03-30T13:07:55.720689+0000","last_undegraded":"2024-03-30T13:07:55.720689+0000","last_fullsized":"2024-03-30T13:07:55.720689+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_clean_scrub_stamp":"2024-03-30T13:07:40.613260+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:54.625356+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":56974905,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":319,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.117216+0000","last_change":"2024-03-30T13:07:08.403925+0000","last_active":"2024-03-30T13:07:58.117216+0000","last_peered":"2024-03-30T13:07:58.117216+0000","last_clean":"2024-03-30T13:07:58.117216+0000","last_became_active":"2024-03-30T13:06:54.155351+0000","last_became_peered":"2024-03-30T13:06:54.155351+0000","last_unstale":"2024-03-30T13:07:58.117216+0000","last_undegraded":"2024-03-30T13:07:58.117216+0000","last_fullsized":"2024-03-30T13:07:58.117216+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:07:08.403663+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:07:08.403663+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:24.852917+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":806,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-30T13:07:57.114051+0000","last_change":"2024-03-30T13:07:45.157507+0000","last_active":"2024-03-30T13:07:57.114051+0000","last_peered":"2024-03-30T13:07:57.114051+0000","last_clean":"2024-03-30T13:07:57.114051+0000","last_became_active":"2024-03-30T13:07:45.157086+0000","last_became_peered":"2024-03-30T13:07:45.157086+0000","last_unstale":"2024-03-30T13:07:57.114051+0000","last_undegraded":"2024-03-30T13:07:57.114051+0000","last_fullsized":"2024-03-30T13:07:57.114051+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:07:44.108131+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:07:44.108131+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:57.687512+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":51411920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":871,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-30T13:07:55.720686+0000","last_change":"2024-03-30T13:07:45.156994+0000","last_active":"2024-03-30T13:07:55.720686+0000","last_peered":"2024-03-30T13:07:55.720686+0000","last_clean":"2024-03-30T13:07:55.720686+0000","last_became_active":"2024-03-30T13:07:45.156128+0000","last_became_peered":"2024-03-30T13:07:45.156128+0000","last_unstale":"2024-03-30T13:07:55.720686+0000","last_undegraded":"2024-03-30T13:07:55.720686+0000","last_fullsized":"2024-03-30T13:07:55.720686+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:05.491728+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:40.339270+0000","last_clean_scrub_stamp":"2024-03-30T13:07:05.491728+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:28.374332+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":65983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":603,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-30T13:07:56.270933+0000","last_change":"2024-03-30T13:06:34.924067+0000","last_active":"2024-03-30T13:07:56.270933+0000","last_peered":"2024-03-30T13:07:56.270933+0000","last_clean":"2024-03-30T13:07:56.270933+0000","last_became_active":"2024-03-30T13:05:07.445793+0000","last_became_peered":"2024-03-30T13:05:07.445793+0000","last_unstale":"2024-03-30T13:07:56.270933+0000","last_undegraded":"2024-03-30T13:07:56.270933+0000","last_fullsized":"2024-03-30T13:07:56.270933+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:06:34.923827+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:06:34.923827+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:01.504665+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":769,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.290641+0000","last_change":"2024-03-30T13:07:48.693563+0000","last_active":"2024-03-30T13:07:58.290641+0000","last_peered":"2024-03-30T13:07:58.290641+0000","last_clean":"2024-03-30T13:07:58.290641+0000","last_became_active":"2024-03-30T13:05:07.445475+0000","last_became_peered":"2024-03-30T13:05:07.445475+0000","last_unstale":"2024-03-30T13:07:58.290641+0000","last_undegraded":"2024-03-30T13:07:58.290641+0000","last_fullsized":"2024-03-30T13:07:58.290641+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:07:48.693155+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:49.070756+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":818,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.290775+0000","last_change":"2024-03-30T13:07:45.158029+0000","last_active":"2024-03-30T13:07:58.290775+0000","last_peered":"2024-03-30T13:07:58.290775+0000","last_clean":"2024-03-30T13:07:58.290775+0000","last_became_active":"2024-03-30T13:07:45.156745+0000","last_became_peered":"2024-03-30T13:07:45.156745+0000","last_unstale":"2024-03-30T13:07:58.290775+0000","last_undegraded":"2024-03-30T13:07:58.290775+0000","last_fullsized":"2024-03-30T13:07:58.290775+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:07:24.693213+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:07:24.693213+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:38.979624+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":57731253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":108,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.765032+0000","last_change":"2024-03-30T13:07:53.686045+0000","last_active":"2024-03-30T13:07:58.765032+0000","last_peered":"2024-03-30T13:07:58.765032+0000","last_clean":"2024-03-30T13:07:58.765032+0000","last_became_active":"2024-03-30T13:07:45.154855+0000","last_became_peered":"2024-03-30T13:07:45.154855+0000","last_unstale":"2024-03-30T13:07:58.765032+0000","last_undegraded":"2024-03-30T13:07:58.765032+0000","last_fullsized":"2024-03-30T13:07:58.765032+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:07:53.685778+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:58.028417+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":257,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.117211+0000","last_change":"2024-03-30T13:07:01.439676+0000","last_active":"2024-03-30T13:07:58.117211+0000","last_peered":"2024-03-30T13:07:58.117211+0000","last_clean":"2024-03-30T13:07:58.117211+0000","last_became_active":"2024-03-30T13:06:54.153682+0000","last_became_peered":"2024-03-30T13:06:54.153682+0000","last_unstale":"2024-03-30T13:07:58.117211+0000","last_undegraded":"2024-03-30T13:07:58.117211+0000","last_fullsized":"2024-03-30T13:07:58.117211+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:07:01.439296+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:07:01.439296+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:21.420094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":108,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.765620+0000","last_change":"2024-03-30T13:07:45.160657+0000","last_active":"2024-03-30T13:07:58.765620+0000","last_peered":"2024-03-30T13:07:58.765620+0000","last_clean":"2024-03-30T13:07:58.765620+0000","last_became_active":"2024-03-30T13:07:45.154381+0000","last_became_peered":"2024-03-30T13:07:45.154381+0000","last_unstale":"2024-03-30T13:07:58.765620+0000","last_undegraded":"2024-03-30T13:07:58.765620+0000","last_fullsized":"2024-03-30T13:07:58.765620+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:06:54.714864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:06:54.714864+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:03.640992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":766,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-30T13:07:57.114160+0000","last_change":"2024-03-30T13:07:45.158087+0000","last_active":"2024-03-30T13:07:57.114160+0000","last_peered":"2024-03-30T13:07:57.114160+0000","last_clean":"2024-03-30T13:07:57.114160+0000","last_became_active":"2024-03-30T13:07:45.157528+0000","last_became_peered":"2024-03-30T13:07:45.157528+0000","last_unstale":"2024-03-30T13:07:57.114160+0000","last_undegraded":"2024-03-30T13:07:57.114160+0000","last_fullsized":"2024-03-30T13:07:57.114160+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:38.221640+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:07:38.221640+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:42.757729+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":26134555,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":318370678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":227,"seq":974957576240,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50764,"kb_used_data":19068,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321076,"statfs":{"total":96636764160,"available":96584781824,"internally_reserved":0,"allocated":19525632,"data_stored":17287999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534725,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56376,"kb_used_data":23788,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94315464,"statfs":{"total":96636764160,"available":96579035136,"internally_reserved":0,"allocated":24358912,"data_stored":22134831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":33350120},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824036,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48076,"kb_used_data":15480,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94323764,"statfs":{"total":96636764160,"available":96587534336,"internally_reserved":0,"allocated":15851520,"data_stored":13634240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509593,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41472,"kb_used_data":10284,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330368,"statfs":{"total":96636764160,"available":96594296832,"internally_reserved":0,"allocated":10530816,"data_stored":8293183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285678,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40784,"kb_used_data":9092,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94331056,"statfs":{"total":96636764160,"available":96595001344,"internally_reserved":0,"allocated":9310208,"data_stored":7092928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509593,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48648,"kb_used_data":17460,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323192,"statfs":{"total":96636764160,"available":96586948608,"internally_reserved":0,"allocated":17879040,"data_stored":15641407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":298,"seq":1279900254223,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47456,"kb_used_data":13968,"kb_used_omap":10,"kb_used_meta":33461,"kb_avail":94324384,"statfs":{"total":96636764160,"available":96588169216,"internally_reserved":0,"allocated":14303232,"data_stored":12085952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10986,"internal_metadata":34264342},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575001,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47612,"kb_used_data":16424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324228,"statfs":{"total":96636764160,"available":96588009472,"internally_reserved":0,"allocated":16818176,"data_stored":14600896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:08:00.636 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:08:00.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:00.749+0000 7fbef7d44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:00.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:00.765+0000 7fbef7d44640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:00.939 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:08:00.940 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:08:00.957 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":703,"stamp":"2024-03-30T13:08:00.299408+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":318370678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":381188,"kb_used_data":125564,"kb_used_omap":39,"kb_used_meta":255448,"kb_avail":754593532,"statfs":{"total":773094113280,"available":772703776768,"internally_reserved":0,"allocated":128577536,"data_stored":110771436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39968,"internal_metadata":261579744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.030276"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":764,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.290859+0000","last_change":"2024-03-30T13:07:45.158080+0000","last_active":"2024-03-30T13:07:58.290859+0000","last_peered":"2024-03-30T13:07:58.290859+0000","last_clean":"2024-03-30T13:07:58.290859+0000","last_became_active":"2024-03-30T13:07:45.156785+0000","last_became_peered":"2024-03-30T13:07:45.156785+0000","last_unstale":"2024-03-30T13:07:58.290859+0000","last_undegraded":"2024-03-30T13:07:58.290859+0000","last_fullsized":"2024-03-30T13:07:58.290859+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:38.717555+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:07:38.717555+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:57.207615+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22941225,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":701,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-30T13:07:56.271016+0000","last_change":"2024-03-30T13:07:26.626194+0000","last_active":"2024-03-30T13:07:56.271016+0000","last_peered":"2024-03-30T13:07:56.271016+0000","last_clean":"2024-03-30T13:07:56.271016+0000","last_became_active":"2024-03-30T13:06:54.153354+0000","last_became_peered":"2024-03-30T13:06:54.153354+0000","last_unstale":"2024-03-30T13:07:56.271016+0000","last_undegraded":"2024-03-30T13:07:56.271016+0000","last_fullsized":"2024-03-30T13:07:56.271016+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:07:26.625854+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:07:26.625854+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:30.881671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":305,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.117258+0000","last_change":"2024-03-30T13:07:23.684003+0000","last_active":"2024-03-30T13:07:58.117258+0000","last_peered":"2024-03-30T13:07:58.117258+0000","last_clean":"2024-03-30T13:07:58.117258+0000","last_became_active":"2024-03-30T13:06:54.154160+0000","last_became_peered":"2024-03-30T13:06:54.154160+0000","last_unstale":"2024-03-30T13:07:58.117258+0000","last_undegraded":"2024-03-30T13:07:58.117258+0000","last_fullsized":"2024-03-30T13:07:58.117258+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:07:23.683745+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_clean_scrub_stamp":"2024-03-30T13:07:23.683745+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:49.948436+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":292,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.117155+0000","last_change":"2024-03-30T13:07:16.522990+0000","last_active":"2024-03-30T13:07:58.117155+0000","last_peered":"2024-03-30T13:07:58.117155+0000","last_clean":"2024-03-30T13:07:58.117155+0000","last_became_active":"2024-03-30T13:06:54.156927+0000","last_became_peered":"2024-03-30T13:06:54.156927+0000","last_unstale":"2024-03-30T13:07:58.117155+0000","last_undegraded":"2024-03-30T13:07:58.117155+0000","last_fullsized":"2024-03-30T13:07:58.117155+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:16.522745+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:07:16.522745+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:40.030402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":92,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.765104+0000","last_change":"2024-03-30T13:07:45.158822+0000","last_active":"2024-03-30T13:07:58.765104+0000","last_peered":"2024-03-30T13:07:58.765104+0000","last_clean":"2024-03-30T13:07:58.765104+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:07:58.765104+0000","last_undegraded":"2024-03-30T13:07:58.765104+0000","last_fullsized":"2024-03-30T13:07:58.765104+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:07:37.576416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:07:37.576416+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:50.550338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":801,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-30T13:07:55.720689+0000","last_change":"2024-03-30T13:07:45.157963+0000","last_active":"2024-03-30T13:07:55.720689+0000","last_peered":"2024-03-30T13:07:55.720689+0000","last_clean":"2024-03-30T13:07:55.720689+0000","last_became_active":"2024-03-30T13:07:45.156174+0000","last_became_peered":"2024-03-30T13:07:45.156174+0000","last_unstale":"2024-03-30T13:07:55.720689+0000","last_undegraded":"2024-03-30T13:07:55.720689+0000","last_fullsized":"2024-03-30T13:07:55.720689+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_clean_scrub_stamp":"2024-03-30T13:07:40.613260+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:54.625356+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":56974905,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":319,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.117216+0000","last_change":"2024-03-30T13:07:08.403925+0000","last_active":"2024-03-30T13:07:58.117216+0000","last_peered":"2024-03-30T13:07:58.117216+0000","last_clean":"2024-03-30T13:07:58.117216+0000","last_became_active":"2024-03-30T13:06:54.155351+0000","last_became_peered":"2024-03-30T13:06:54.155351+0000","last_unstale":"2024-03-30T13:07:58.117216+0000","last_undegraded":"2024-03-30T13:07:58.117216+0000","last_fullsized":"2024-03-30T13:07:58.117216+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:07:08.403663+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:07:08.403663+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:24.852917+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":806,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-30T13:07:57.114051+0000","last_change":"2024-03-30T13:07:45.157507+0000","last_active":"2024-03-30T13:07:57.114051+0000","last_peered":"2024-03-30T13:07:57.114051+0000","last_clean":"2024-03-30T13:07:57.114051+0000","last_became_active":"2024-03-30T13:07:45.157086+0000","last_became_peered":"2024-03-30T13:07:45.157086+0000","last_unstale":"2024-03-30T13:07:57.114051+0000","last_undegraded":"2024-03-30T13:07:57.114051+0000","last_fullsized":"2024-03-30T13:07:57.114051+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:07:44.108131+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:07:44.108131+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:57.687512+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":51411920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":871,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-30T13:07:55.720686+0000","last_change":"2024-03-30T13:07:45.156994+0000","last_active":"2024-03-30T13:07:55.720686+0000","last_peered":"2024-03-30T13:07:55.720686+0000","last_clean":"2024-03-30T13:07:55.720686+0000","last_became_active":"2024-03-30T13:07:45.156128+0000","last_became_peered":"2024-03-30T13:07:45.156128+0000","last_unstale":"2024-03-30T13:07:55.720686+0000","last_undegraded":"2024-03-30T13:07:55.720686+0000","last_fullsized":"2024-03-30T13:07:55.720686+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:05.491728+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:40.339270+0000","last_clean_scrub_stamp":"2024-03-30T13:07:05.491728+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:28.374332+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":65983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":603,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-30T13:07:56.270933+0000","last_change":"2024-03-30T13:06:34.924067+0000","last_active":"2024-03-30T13:07:56.270933+0000","last_peered":"2024-03-30T13:07:56.270933+0000","last_clean":"2024-03-30T13:07:56.270933+0000","last_became_active":"2024-03-30T13:05:07.445793+0000","last_became_peered":"2024-03-30T13:05:07.445793+0000","last_unstale":"2024-03-30T13:07:56.270933+0000","last_undegraded":"2024-03-30T13:07:56.270933+0000","last_fullsized":"2024-03-30T13:07:56.270933+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:06:34.923827+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:06:34.923827+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:01.504665+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":769,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.290641+0000","last_change":"2024-03-30T13:07:48.693563+0000","last_active":"2024-03-30T13:07:58.290641+0000","last_peered":"2024-03-30T13:07:58.290641+0000","last_clean":"2024-03-30T13:07:58.290641+0000","last_became_active":"2024-03-30T13:05:07.445475+0000","last_became_peered":"2024-03-30T13:05:07.445475+0000","last_unstale":"2024-03-30T13:07:58.290641+0000","last_undegraded":"2024-03-30T13:07:58.290641+0000","last_fullsized":"2024-03-30T13:07:58.290641+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:07:48.693155+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:49.070756+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":818,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.290775+0000","last_change":"2024-03-30T13:07:45.158029+0000","last_active":"2024-03-30T13:07:58.290775+0000","last_peered":"2024-03-30T13:07:58.290775+0000","last_clean":"2024-03-30T13:07:58.290775+0000","last_became_active":"2024-03-30T13:07:45.156745+0000","last_became_peered":"2024-03-30T13:07:45.156745+0000","last_unstale":"2024-03-30T13:07:58.290775+0000","last_undegraded":"2024-03-30T13:07:58.290775+0000","last_fullsized":"2024-03-30T13:07:58.290775+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:07:24.693213+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:07:24.693213+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:38.979624+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":57731253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":108,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.765032+0000","last_change":"2024-03-30T13:07:53.686045+0000","last_active":"2024-03-30T13:07:58.765032+0000","last_peered":"2024-03-30T13:07:58.765032+0000","last_clean":"2024-03-30T13:07:58.765032+0000","last_became_active":"2024-03-30T13:07:45.154855+0000","last_became_peered":"2024-03-30T13:07:45.154855+0000","last_unstale":"2024-03-30T13:07:58.765032+0000","last_undegraded":"2024-03-30T13:07:58.765032+0000","last_fullsized":"2024-03-30T13:07:58.765032+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:07:53.685778+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:58.028417+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":257,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.117211+0000","last_change":"2024-03-30T13:07:01.439676+0000","last_active":"2024-03-30T13:07:58.117211+0000","last_peered":"2024-03-30T13:07:58.117211+0000","last_clean":"2024-03-30T13:07:58.117211+0000","last_became_active":"2024-03-30T13:06:54.153682+0000","last_became_peered":"2024-03-30T13:06:54.153682+0000","last_unstale":"2024-03-30T13:07:58.117211+0000","last_undegraded":"2024-03-30T13:07:58.117211+0000","last_fullsized":"2024-03-30T13:07:58.117211+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:07:01.439296+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:07:01.439296+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:21.420094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":108,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.765620+0000","last_change":"2024-03-30T13:07:45.160657+0000","last_active":"2024-03-30T13:07:58.765620+0000","last_peered":"2024-03-30T13:07:58.765620+0000","last_clean":"2024-03-30T13:07:58.765620+0000","last_became_active":"2024-03-30T13:07:45.154381+0000","last_became_peered":"2024-03-30T13:07:45.154381+0000","last_unstale":"2024-03-30T13:07:58.765620+0000","last_undegraded":"2024-03-30T13:07:58.765620+0000","last_fullsized":"2024-03-30T13:07:58.765620+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:06:54.714864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:06:54.714864+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:03.640992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":766,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-30T13:07:57.114160+0000","last_change":"2024-03-30T13:07:45.158087+0000","last_active":"2024-03-30T13:07:57.114160+0000","last_peered":"2024-03-30T13:07:57.114160+0000","last_clean":"2024-03-30T13:07:57.114160+0000","last_became_active":"2024-03-30T13:07:45.157528+0000","last_became_peered":"2024-03-30T13:07:45.157528+0000","last_unstale":"2024-03-30T13:07:57.114160+0000","last_undegraded":"2024-03-30T13:07:57.114160+0000","last_fullsized":"2024-03-30T13:07:57.114160+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:38.221640+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:07:38.221640+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:42.757729+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":26134555,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":318370678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":227,"seq":974957576240,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50764,"kb_used_data":19068,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321076,"statfs":{"total":96636764160,"available":96584781824,"internally_reserved":0,"allocated":19525632,"data_stored":17287999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534725,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56376,"kb_used_data":23788,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94315464,"statfs":{"total":96636764160,"available":96579035136,"internally_reserved":0,"allocated":24358912,"data_stored":22134831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":33350120},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824036,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48076,"kb_used_data":15480,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94323764,"statfs":{"total":96636764160,"available":96587534336,"internally_reserved":0,"allocated":15851520,"data_stored":13634240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509593,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41472,"kb_used_data":10284,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330368,"statfs":{"total":96636764160,"available":96594296832,"internally_reserved":0,"allocated":10530816,"data_stored":8293183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285678,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40784,"kb_used_data":9092,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94331056,"statfs":{"total":96636764160,"available":96595001344,"internally_reserved":0,"allocated":9310208,"data_stored":7092928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509593,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48648,"kb_used_data":17460,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323192,"statfs":{"total":96636764160,"available":96586948608,"internally_reserved":0,"allocated":17879040,"data_stored":15641407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":298,"seq":1279900254223,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47456,"kb_used_data":13968,"kb_used_omap":10,"kb_used_meta":33461,"kb_avail":94324384,"statfs":{"total":96636764160,"available":96588169216,"internally_reserved":0,"allocated":14303232,"data_stored":12085952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10986,"internal_metadata":34264342},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575001,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47612,"kb_used_data":16424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324228,"statfs":{"total":96636764160,"available":96588009472,"internally_reserved":0,"allocated":16818176,"data_stored":14600896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:08:00.959 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:08:00.959 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:08:01.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:01.081+0000 7f23d3cd5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:01.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:01.093+0000 7f23d3cd5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:01.260 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:08:01.260 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":336,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:08:00.296705+0000","last_up_change":"2024-03-30T13:07:44.122755+0000","last_in_change":"2024-03-30T13:07:44.122755+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":19,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":77,"up_from":83,"up_thru":320,"down_at":78,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6818","nonce":3104224893}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6821","nonce":3104224893}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6828","nonce":3104224893}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":3104224893},{"type":"v1","addr":"172.21.15.67:6825","nonce":3104224893}]},"public_addr":"172.21.15.67:6818/3104224893","cluster_addr":"172.21.15.67:6821/3104224893","heartbeat_back_addr":"172.21.15.67:6828/3104224893","heartbeat_front_addr":"172.21.15.67:6825/3104224893","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":315,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":320,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6801","nonce":1504348724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6803","nonce":1504348724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6807","nonce":1504348724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6805","nonce":1504348724}]},"public_addr":"172.21.15.67:6801/1504348724","cluster_addr":"172.21.15.67:6803/1504348724","heartbeat_back_addr":"172.21.15.67:6807/1504348724","heartbeat_front_addr":"172.21.15.67:6805/1504348724","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":135,"last_clean_end":243,"up_from":250,"up_thru":316,"down_at":244,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6803","nonce":3986522908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6807","nonce":3986522908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6815","nonce":3986522908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6811","nonce":3986522908}]},"public_addr":"172.21.15.82:6803/3986522908","cluster_addr":"172.21.15.82:6807/3986522908","heartbeat_back_addr":"172.21.15.82:6815/3986522908","heartbeat_front_addr":"172.21.15.82:6811/3986522908","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":274,"last_clean_end":314,"up_from":320,"up_thru":320,"down_at":315,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":925466951},{"type":"v1","addr":"172.21.15.82:6817","nonce":925466951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":925466951},{"type":"v1","addr":"172.21.15.82:6819","nonce":925466951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":925466951},{"type":"v1","addr":"172.21.15.82:6823","nonce":925466951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":925466951},{"type":"v1","addr":"172.21.15.82:6821","nonce":925466951}]},"public_addr":"172.21.15.82:6817/925466951","cluster_addr":"172.21.15.82:6819/925466951","heartbeat_back_addr":"172.21.15.82:6823/925466951","heartbeat_front_addr":"172.21.15.82:6821/925466951","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":222,"up_from":227,"up_thru":320,"down_at":223,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3068584906},{"type":"v1","addr":"172.21.15.82:6801","nonce":3068584906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3068584906},{"type":"v1","addr":"172.21.15.82:6806","nonce":3068584906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3068584906},{"type":"v1","addr":"172.21.15.82:6814","nonce":3068584906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3068584906},{"type":"v1","addr":"172.21.15.82:6810","nonce":3068584906}]},"public_addr":"172.21.15.82:6801/3068584906","cluster_addr":"172.21.15.82:6806/3068584906","heartbeat_back_addr":"172.21.15.82:6814/3068584906","heartbeat_front_addr":"172.21.15.82:6810/3068584906","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":206,"last_clean_end":292,"up_from":298,"up_thru":298,"down_at":293,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3894469331},{"type":"v1","addr":"172.21.15.82:6825","nonce":3894469331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3894469331},{"type":"v1","addr":"172.21.15.82:6827","nonce":3894469331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3894469331},{"type":"v1","addr":"172.21.15.82:6831","nonce":3894469331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3894469331},{"type":"v1","addr":"172.21.15.82:6829","nonce":3894469331}]},"public_addr":"172.21.15.82:6825/3894469331","cluster_addr":"172.21.15.82:6827/3894469331","heartbeat_back_addr":"172.21.15.82:6831/3894469331","heartbeat_front_addr":"172.21.15.82:6829/3894469331","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:07:04.796993+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:07:06.436617+0000","dead_epoch":78},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:07:57.620702+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:07:14.138322+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:06:57.279609+0000","dead_epoch":244},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:07:03.994444+0000","dead_epoch":315},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:06:44.638787+0000","dead_epoch":223},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:06:52.464189+0000","dead_epoch":293}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:08:01.278 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:08:01.278 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:08:01.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:01.385+0000 7fa59f0cf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:01.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:01.405+0000 7fa59f0cf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:01.570 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:08:01.589 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:08:01.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:01.709+0000 7f022839b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:01.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:01.725+0000 7f022839b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:01.901 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:08:01.901 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:08:01.919 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:08:01.919 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:08:01.919 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:08:01.919 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:08:01.919 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 3, 1, 6, 4, 7, 5] 2024-03-30T13:08:01.919 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:08:01.919 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:08:02.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:02.033+0000 7ffb28821640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:02.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:02.045+0000 7ffb28821640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:02.222 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:08:02.223 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:08:02.241 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":704,"stamp":"2024-03-30T13:08:01.996270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":318370678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":381240,"kb_used_data":125624,"kb_used_omap":39,"kb_used_meta":255448,"kb_avail":754593480,"statfs":{"total":773094113280,"available":772703723520,"internally_reserved":0,"allocated":128638976,"data_stored":110798921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39968,"internal_metadata":261579744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.724422"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":764,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.290859+0000","last_change":"2024-03-30T13:07:45.158080+0000","last_active":"2024-03-30T13:07:58.290859+0000","last_peered":"2024-03-30T13:07:58.290859+0000","last_clean":"2024-03-30T13:07:58.290859+0000","last_became_active":"2024-03-30T13:07:45.156785+0000","last_became_peered":"2024-03-30T13:07:45.156785+0000","last_unstale":"2024-03-30T13:07:58.290859+0000","last_undegraded":"2024-03-30T13:07:58.290859+0000","last_fullsized":"2024-03-30T13:07:58.290859+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:38.717555+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:07:38.717555+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:57.207615+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22941225,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":709,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-30T13:08:00.301516+0000","last_change":"2024-03-30T13:07:26.626194+0000","last_active":"2024-03-30T13:08:00.301516+0000","last_peered":"2024-03-30T13:08:00.301516+0000","last_clean":"2024-03-30T13:08:00.301516+0000","last_became_active":"2024-03-30T13:06:54.153354+0000","last_became_peered":"2024-03-30T13:06:54.153354+0000","last_unstale":"2024-03-30T13:08:00.301516+0000","last_undegraded":"2024-03-30T13:08:00.301516+0000","last_fullsized":"2024-03-30T13:08:00.301516+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:07:26.625854+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:07:26.625854+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:30.881671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":305,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.117258+0000","last_change":"2024-03-30T13:07:23.684003+0000","last_active":"2024-03-30T13:07:58.117258+0000","last_peered":"2024-03-30T13:07:58.117258+0000","last_clean":"2024-03-30T13:07:58.117258+0000","last_became_active":"2024-03-30T13:06:54.154160+0000","last_became_peered":"2024-03-30T13:06:54.154160+0000","last_unstale":"2024-03-30T13:07:58.117258+0000","last_undegraded":"2024-03-30T13:07:58.117258+0000","last_fullsized":"2024-03-30T13:07:58.117258+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:07:23.683745+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_clean_scrub_stamp":"2024-03-30T13:07:23.683745+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:49.948436+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":292,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.117155+0000","last_change":"2024-03-30T13:07:16.522990+0000","last_active":"2024-03-30T13:07:58.117155+0000","last_peered":"2024-03-30T13:07:58.117155+0000","last_clean":"2024-03-30T13:07:58.117155+0000","last_became_active":"2024-03-30T13:06:54.156927+0000","last_became_peered":"2024-03-30T13:06:54.156927+0000","last_unstale":"2024-03-30T13:07:58.117155+0000","last_undegraded":"2024-03-30T13:07:58.117155+0000","last_fullsized":"2024-03-30T13:07:58.117155+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:16.522745+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:07:16.522745+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:40.030402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":92,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.765104+0000","last_change":"2024-03-30T13:07:45.158822+0000","last_active":"2024-03-30T13:07:58.765104+0000","last_peered":"2024-03-30T13:07:58.765104+0000","last_clean":"2024-03-30T13:07:58.765104+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:07:58.765104+0000","last_undegraded":"2024-03-30T13:07:58.765104+0000","last_fullsized":"2024-03-30T13:07:58.765104+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:07:37.576416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:07:37.576416+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:50.550338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":801,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-30T13:07:55.720689+0000","last_change":"2024-03-30T13:07:45.157963+0000","last_active":"2024-03-30T13:07:55.720689+0000","last_peered":"2024-03-30T13:07:55.720689+0000","last_clean":"2024-03-30T13:07:55.720689+0000","last_became_active":"2024-03-30T13:07:45.156174+0000","last_became_peered":"2024-03-30T13:07:45.156174+0000","last_unstale":"2024-03-30T13:07:55.720689+0000","last_undegraded":"2024-03-30T13:07:55.720689+0000","last_fullsized":"2024-03-30T13:07:55.720689+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_clean_scrub_stamp":"2024-03-30T13:07:40.613260+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:54.625356+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":56974905,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":319,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.117216+0000","last_change":"2024-03-30T13:07:08.403925+0000","last_active":"2024-03-30T13:07:58.117216+0000","last_peered":"2024-03-30T13:07:58.117216+0000","last_clean":"2024-03-30T13:07:58.117216+0000","last_became_active":"2024-03-30T13:06:54.155351+0000","last_became_peered":"2024-03-30T13:06:54.155351+0000","last_unstale":"2024-03-30T13:07:58.117216+0000","last_undegraded":"2024-03-30T13:07:58.117216+0000","last_fullsized":"2024-03-30T13:07:58.117216+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:07:08.403663+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:07:08.403663+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:24.852917+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":806,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-30T13:07:57.114051+0000","last_change":"2024-03-30T13:07:45.157507+0000","last_active":"2024-03-30T13:07:57.114051+0000","last_peered":"2024-03-30T13:07:57.114051+0000","last_clean":"2024-03-30T13:07:57.114051+0000","last_became_active":"2024-03-30T13:07:45.157086+0000","last_became_peered":"2024-03-30T13:07:45.157086+0000","last_unstale":"2024-03-30T13:07:57.114051+0000","last_undegraded":"2024-03-30T13:07:57.114051+0000","last_fullsized":"2024-03-30T13:07:57.114051+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:07:44.108131+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:07:44.108131+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:57.687512+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":51411920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":871,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-30T13:07:55.720686+0000","last_change":"2024-03-30T13:07:45.156994+0000","last_active":"2024-03-30T13:07:55.720686+0000","last_peered":"2024-03-30T13:07:55.720686+0000","last_clean":"2024-03-30T13:07:55.720686+0000","last_became_active":"2024-03-30T13:07:45.156128+0000","last_became_peered":"2024-03-30T13:07:45.156128+0000","last_unstale":"2024-03-30T13:07:55.720686+0000","last_undegraded":"2024-03-30T13:07:55.720686+0000","last_fullsized":"2024-03-30T13:07:55.720686+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:05.491728+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:40.339270+0000","last_clean_scrub_stamp":"2024-03-30T13:07:05.491728+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:28.374332+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":65983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":611,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-30T13:08:00.301325+0000","last_change":"2024-03-30T13:06:34.924067+0000","last_active":"2024-03-30T13:08:00.301325+0000","last_peered":"2024-03-30T13:08:00.301325+0000","last_clean":"2024-03-30T13:08:00.301325+0000","last_became_active":"2024-03-30T13:05:07.445793+0000","last_became_peered":"2024-03-30T13:05:07.445793+0000","last_unstale":"2024-03-30T13:08:00.301325+0000","last_undegraded":"2024-03-30T13:08:00.301325+0000","last_fullsized":"2024-03-30T13:08:00.301325+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:06:34.923827+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:06:34.923827+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:01.504665+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":769,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.290641+0000","last_change":"2024-03-30T13:07:48.693563+0000","last_active":"2024-03-30T13:07:58.290641+0000","last_peered":"2024-03-30T13:07:58.290641+0000","last_clean":"2024-03-30T13:07:58.290641+0000","last_became_active":"2024-03-30T13:05:07.445475+0000","last_became_peered":"2024-03-30T13:05:07.445475+0000","last_unstale":"2024-03-30T13:07:58.290641+0000","last_undegraded":"2024-03-30T13:07:58.290641+0000","last_fullsized":"2024-03-30T13:07:58.290641+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:07:48.693155+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:49.070756+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":818,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.290775+0000","last_change":"2024-03-30T13:07:45.158029+0000","last_active":"2024-03-30T13:07:58.290775+0000","last_peered":"2024-03-30T13:07:58.290775+0000","last_clean":"2024-03-30T13:07:58.290775+0000","last_became_active":"2024-03-30T13:07:45.156745+0000","last_became_peered":"2024-03-30T13:07:45.156745+0000","last_unstale":"2024-03-30T13:07:58.290775+0000","last_undegraded":"2024-03-30T13:07:58.290775+0000","last_fullsized":"2024-03-30T13:07:58.290775+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:07:24.693213+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:07:24.693213+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:38.979624+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":57731253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":108,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.765032+0000","last_change":"2024-03-30T13:07:53.686045+0000","last_active":"2024-03-30T13:07:58.765032+0000","last_peered":"2024-03-30T13:07:58.765032+0000","last_clean":"2024-03-30T13:07:58.765032+0000","last_became_active":"2024-03-30T13:07:45.154855+0000","last_became_peered":"2024-03-30T13:07:45.154855+0000","last_unstale":"2024-03-30T13:07:58.765032+0000","last_undegraded":"2024-03-30T13:07:58.765032+0000","last_fullsized":"2024-03-30T13:07:58.765032+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:07:53.685778+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:58.028417+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":257,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.117211+0000","last_change":"2024-03-30T13:07:01.439676+0000","last_active":"2024-03-30T13:07:58.117211+0000","last_peered":"2024-03-30T13:07:58.117211+0000","last_clean":"2024-03-30T13:07:58.117211+0000","last_became_active":"2024-03-30T13:06:54.153682+0000","last_became_peered":"2024-03-30T13:06:54.153682+0000","last_unstale":"2024-03-30T13:07:58.117211+0000","last_undegraded":"2024-03-30T13:07:58.117211+0000","last_fullsized":"2024-03-30T13:07:58.117211+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:07:01.439296+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:07:01.439296+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:21.420094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":108,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.765620+0000","last_change":"2024-03-30T13:07:45.160657+0000","last_active":"2024-03-30T13:07:58.765620+0000","last_peered":"2024-03-30T13:07:58.765620+0000","last_clean":"2024-03-30T13:07:58.765620+0000","last_became_active":"2024-03-30T13:07:45.154381+0000","last_became_peered":"2024-03-30T13:07:45.154381+0000","last_unstale":"2024-03-30T13:07:58.765620+0000","last_undegraded":"2024-03-30T13:07:58.765620+0000","last_fullsized":"2024-03-30T13:07:58.765620+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:06:54.714864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:06:54.714864+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:03.640992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":766,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-30T13:07:57.114160+0000","last_change":"2024-03-30T13:07:45.158087+0000","last_active":"2024-03-30T13:07:57.114160+0000","last_peered":"2024-03-30T13:07:57.114160+0000","last_clean":"2024-03-30T13:07:57.114160+0000","last_became_active":"2024-03-30T13:07:45.157528+0000","last_became_peered":"2024-03-30T13:07:45.157528+0000","last_unstale":"2024-03-30T13:07:57.114160+0000","last_undegraded":"2024-03-30T13:07:57.114160+0000","last_fullsized":"2024-03-30T13:07:57.114160+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:38.221640+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:07:38.221640+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:42.757729+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":26134555,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":318370678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":227,"seq":974957576240,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50764,"kb_used_data":19068,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321076,"statfs":{"total":96636764160,"available":96584781824,"internally_reserved":0,"allocated":19525632,"data_stored":17287999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534725,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56376,"kb_used_data":23788,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94315464,"statfs":{"total":96636764160,"available":96579035136,"internally_reserved":0,"allocated":24358912,"data_stored":22134831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":33350120},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824037,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48128,"kb_used_data":15540,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94323712,"statfs":{"total":96636764160,"available":96587481088,"internally_reserved":0,"allocated":15912960,"data_stored":13661725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509593,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41472,"kb_used_data":10284,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330368,"statfs":{"total":96636764160,"available":96594296832,"internally_reserved":0,"allocated":10530816,"data_stored":8293183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":83,"seq":356482285678,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40784,"kb_used_data":9092,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94331056,"statfs":{"total":96636764160,"available":96595001344,"internally_reserved":0,"allocated":9310208,"data_stored":7092928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509593,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48648,"kb_used_data":17460,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323192,"statfs":{"total":96636764160,"available":96586948608,"internally_reserved":0,"allocated":17879040,"data_stored":15641407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":298,"seq":1279900254223,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47456,"kb_used_data":13968,"kb_used_omap":10,"kb_used_meta":33461,"kb_avail":94324384,"statfs":{"total":96636764160,"available":96588169216,"internally_reserved":0,"allocated":14303232,"data_stored":12085952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10986,"internal_metadata":34264342},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575001,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47612,"kb_used_data":16424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324228,"statfs":{"total":96636764160,"available":96588009472,"internally_reserved":0,"allocated":16818176,"data_stored":14600896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:08:02.242 INFO:tasks.thrashosds.thrasher:Choosing PG 1.3 with acting set [1, 6, 5] 2024-03-30T13:08:02.242 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 2, 3, 1, 6, 4, 7, 5] 2024-03-30T13:08:02.242 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-30T13:08:02.242 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:08:02.313 INFO:tasks.ceph.osd.1:Stopped 2024-03-30T13:08:02.313 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 2, 3, 1, 6, 4, 7, 5] 2024-03-30T13:08:02.314 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 1 2024-03-30T13:08:02.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:02.433+0000 7ff355f00640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:02.449+0000 7ff355f00640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:02.631 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:08:02.633+0000 7fbb38f08640 -1 mon.a@0(leader).osd e337 definitely_dead 0 2024-03-30T13:08:03.316 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.1. 2024-03-30T13:08:03.346 INFO:tasks.thrashosds.thrasher:dead_osds=[1], live_osds=[0, 2, 3, 6, 4, 7, 5] 2024-03-30T13:08:03.346 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:08:03.346 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:08:03.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:03.469+0000 7f35c59ca640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:03.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:03.481+0000 7f35c59ca640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:03.655 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:08:03.656 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:08:03.674 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":706,"stamp":"2024-03-30T13:08:03.321687+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":318370678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":45,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":340572,"kb_used_data":116652,"kb_used_omap":34,"kb_used_meta":223773,"kb_avail":660262308,"statfs":{"total":676457349120,"available":676108603392,"internally_reserved":0,"allocated":119451648,"data_stored":103760963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35570,"internal_metadata":229143822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.033217"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":764,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.290859+0000","last_change":"2024-03-30T13:07:45.158080+0000","last_active":"2024-03-30T13:07:58.290859+0000","last_peered":"2024-03-30T13:07:58.290859+0000","last_clean":"2024-03-30T13:07:58.290859+0000","last_became_active":"2024-03-30T13:07:45.156785+0000","last_became_peered":"2024-03-30T13:07:45.156785+0000","last_unstale":"2024-03-30T13:07:58.290859+0000","last_undegraded":"2024-03-30T13:07:58.290859+0000","last_fullsized":"2024-03-30T13:07:58.290859+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:38.717555+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:07:38.717555+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:57.207615+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22941225,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":709,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-30T13:08:00.301516+0000","last_change":"2024-03-30T13:07:26.626194+0000","last_active":"2024-03-30T13:08:00.301516+0000","last_peered":"2024-03-30T13:08:00.301516+0000","last_clean":"2024-03-30T13:08:00.301516+0000","last_became_active":"2024-03-30T13:06:54.153354+0000","last_became_peered":"2024-03-30T13:06:54.153354+0000","last_unstale":"2024-03-30T13:08:00.301516+0000","last_undegraded":"2024-03-30T13:08:00.301516+0000","last_fullsized":"2024-03-30T13:08:00.301516+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:07:26.625854+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:07:26.625854+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:30.881671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":312,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-30T13:08:01.618806+0000","last_change":"2024-03-30T13:07:23.684003+0000","last_active":"2024-03-30T13:08:01.618806+0000","last_peered":"2024-03-30T13:08:01.618806+0000","last_clean":"2024-03-30T13:08:01.618806+0000","last_became_active":"2024-03-30T13:06:54.154160+0000","last_became_peered":"2024-03-30T13:06:54.154160+0000","last_unstale":"2024-03-30T13:08:01.618806+0000","last_undegraded":"2024-03-30T13:08:01.618806+0000","last_fullsized":"2024-03-30T13:08:01.618806+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:07:23.683745+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_clean_scrub_stamp":"2024-03-30T13:07:23.683745+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:49.948436+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":299,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-30T13:08:01.618722+0000","last_change":"2024-03-30T13:07:16.522990+0000","last_active":"2024-03-30T13:08:01.618722+0000","last_peered":"2024-03-30T13:08:01.618722+0000","last_clean":"2024-03-30T13:08:01.618722+0000","last_became_active":"2024-03-30T13:06:54.156927+0000","last_became_peered":"2024-03-30T13:06:54.156927+0000","last_unstale":"2024-03-30T13:08:01.618722+0000","last_undegraded":"2024-03-30T13:08:01.618722+0000","last_fullsized":"2024-03-30T13:08:01.618722+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:16.522745+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:07:16.522745+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:40.030402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":92,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.765104+0000","last_change":"2024-03-30T13:07:45.158822+0000","last_active":"2024-03-30T13:07:58.765104+0000","last_peered":"2024-03-30T13:07:58.765104+0000","last_clean":"2024-03-30T13:07:58.765104+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:07:58.765104+0000","last_undegraded":"2024-03-30T13:07:58.765104+0000","last_fullsized":"2024-03-30T13:07:58.765104+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:07:37.576416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:07:37.576416+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:50.550338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":810,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-30T13:08:00.794170+0000","last_change":"2024-03-30T13:07:45.157963+0000","last_active":"2024-03-30T13:08:00.794170+0000","last_peered":"2024-03-30T13:08:00.794170+0000","last_clean":"2024-03-30T13:08:00.794170+0000","last_became_active":"2024-03-30T13:07:45.156174+0000","last_became_peered":"2024-03-30T13:07:45.156174+0000","last_unstale":"2024-03-30T13:08:00.794170+0000","last_undegraded":"2024-03-30T13:08:00.794170+0000","last_fullsized":"2024-03-30T13:08:00.794170+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_clean_scrub_stamp":"2024-03-30T13:07:40.613260+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:54.625356+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":56974905,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":326,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-30T13:08:01.618729+0000","last_change":"2024-03-30T13:07:08.403925+0000","last_active":"2024-03-30T13:08:01.618729+0000","last_peered":"2024-03-30T13:08:01.618729+0000","last_clean":"2024-03-30T13:08:01.618729+0000","last_became_active":"2024-03-30T13:06:54.155351+0000","last_became_peered":"2024-03-30T13:06:54.155351+0000","last_unstale":"2024-03-30T13:08:01.618729+0000","last_undegraded":"2024-03-30T13:08:01.618729+0000","last_fullsized":"2024-03-30T13:08:01.618729+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:07:08.403663+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:07:08.403663+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:24.852917+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":806,"reported_epoch":332,"state":"stale+active+clean","last_fresh":"2024-03-30T13:07:57.114051+0000","last_change":"2024-03-30T13:07:45.157507+0000","last_active":"2024-03-30T13:07:57.114051+0000","last_peered":"2024-03-30T13:07:57.114051+0000","last_clean":"2024-03-30T13:07:57.114051+0000","last_became_active":"2024-03-30T13:07:45.157086+0000","last_became_peered":"2024-03-30T13:07:45.157086+0000","last_unstale":"2024-03-30T13:08:03.321709+0000","last_undegraded":"2024-03-30T13:07:57.114051+0000","last_fullsized":"2024-03-30T13:07:57.114051+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:07:44.108131+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:07:44.108131+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:57.687512+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":51411920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":880,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-30T13:08:00.794282+0000","last_change":"2024-03-30T13:07:45.156994+0000","last_active":"2024-03-30T13:08:00.794282+0000","last_peered":"2024-03-30T13:08:00.794282+0000","last_clean":"2024-03-30T13:08:00.794282+0000","last_became_active":"2024-03-30T13:07:45.156128+0000","last_became_peered":"2024-03-30T13:07:45.156128+0000","last_unstale":"2024-03-30T13:08:00.794282+0000","last_undegraded":"2024-03-30T13:08:00.794282+0000","last_fullsized":"2024-03-30T13:08:00.794282+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:05.491728+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:40.339270+0000","last_clean_scrub_stamp":"2024-03-30T13:07:05.491728+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:28.374332+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":65983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":611,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-30T13:08:00.301325+0000","last_change":"2024-03-30T13:06:34.924067+0000","last_active":"2024-03-30T13:08:00.301325+0000","last_peered":"2024-03-30T13:08:00.301325+0000","last_clean":"2024-03-30T13:08:00.301325+0000","last_became_active":"2024-03-30T13:05:07.445793+0000","last_became_peered":"2024-03-30T13:05:07.445793+0000","last_unstale":"2024-03-30T13:08:00.301325+0000","last_undegraded":"2024-03-30T13:08:00.301325+0000","last_fullsized":"2024-03-30T13:08:00.301325+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:06:34.923827+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:06:34.923827+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:01.504665+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":769,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.290641+0000","last_change":"2024-03-30T13:07:48.693563+0000","last_active":"2024-03-30T13:07:58.290641+0000","last_peered":"2024-03-30T13:07:58.290641+0000","last_clean":"2024-03-30T13:07:58.290641+0000","last_became_active":"2024-03-30T13:05:07.445475+0000","last_became_peered":"2024-03-30T13:05:07.445475+0000","last_unstale":"2024-03-30T13:07:58.290641+0000","last_undegraded":"2024-03-30T13:07:58.290641+0000","last_fullsized":"2024-03-30T13:07:58.290641+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:07:48.693155+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:49.070756+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":818,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.290775+0000","last_change":"2024-03-30T13:07:45.158029+0000","last_active":"2024-03-30T13:07:58.290775+0000","last_peered":"2024-03-30T13:07:58.290775+0000","last_clean":"2024-03-30T13:07:58.290775+0000","last_became_active":"2024-03-30T13:07:45.156745+0000","last_became_peered":"2024-03-30T13:07:45.156745+0000","last_unstale":"2024-03-30T13:07:58.290775+0000","last_undegraded":"2024-03-30T13:07:58.290775+0000","last_fullsized":"2024-03-30T13:07:58.290775+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:07:24.693213+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:07:24.693213+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:38.979624+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":57731253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":108,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.765032+0000","last_change":"2024-03-30T13:07:53.686045+0000","last_active":"2024-03-30T13:07:58.765032+0000","last_peered":"2024-03-30T13:07:58.765032+0000","last_clean":"2024-03-30T13:07:58.765032+0000","last_became_active":"2024-03-30T13:07:45.154855+0000","last_became_peered":"2024-03-30T13:07:45.154855+0000","last_unstale":"2024-03-30T13:07:58.765032+0000","last_undegraded":"2024-03-30T13:07:58.765032+0000","last_fullsized":"2024-03-30T13:07:58.765032+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:07:53.685778+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:58.028417+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":264,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-30T13:08:01.618728+0000","last_change":"2024-03-30T13:07:01.439676+0000","last_active":"2024-03-30T13:08:01.618728+0000","last_peered":"2024-03-30T13:08:01.618728+0000","last_clean":"2024-03-30T13:08:01.618728+0000","last_became_active":"2024-03-30T13:06:54.153682+0000","last_became_peered":"2024-03-30T13:06:54.153682+0000","last_unstale":"2024-03-30T13:08:01.618728+0000","last_undegraded":"2024-03-30T13:08:01.618728+0000","last_fullsized":"2024-03-30T13:08:01.618728+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:07:01.439296+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:07:01.439296+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:21.420094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":108,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-03-30T13:07:58.765620+0000","last_change":"2024-03-30T13:07:45.160657+0000","last_active":"2024-03-30T13:07:58.765620+0000","last_peered":"2024-03-30T13:07:58.765620+0000","last_clean":"2024-03-30T13:07:58.765620+0000","last_became_active":"2024-03-30T13:07:45.154381+0000","last_became_peered":"2024-03-30T13:07:45.154381+0000","last_unstale":"2024-03-30T13:07:58.765620+0000","last_undegraded":"2024-03-30T13:07:58.765620+0000","last_fullsized":"2024-03-30T13:07:58.765620+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:06:54.714864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:06:54.714864+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:03.640992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":766,"reported_epoch":332,"state":"stale+active+clean","last_fresh":"2024-03-30T13:07:57.114160+0000","last_change":"2024-03-30T13:07:45.158087+0000","last_active":"2024-03-30T13:07:57.114160+0000","last_peered":"2024-03-30T13:07:57.114160+0000","last_clean":"2024-03-30T13:07:57.114160+0000","last_became_active":"2024-03-30T13:07:45.157528+0000","last_became_peered":"2024-03-30T13:07:45.157528+0000","last_unstale":"2024-03-30T13:08:03.321715+0000","last_undegraded":"2024-03-30T13:07:57.114160+0000","last_fullsized":"2024-03-30T13:07:57.114160+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:38.221640+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:07:38.221640+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:42.757729+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":26134555,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":318370678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96309248,"data_stored":96309248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":227,"seq":974957576240,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50764,"kb_used_data":19068,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321076,"statfs":{"total":96636764160,"available":96584781824,"internally_reserved":0,"allocated":19525632,"data_stored":17287999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534725,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56376,"kb_used_data":23788,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94315464,"statfs":{"total":96636764160,"available":96579035136,"internally_reserved":0,"allocated":24358912,"data_stored":22134831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":33350120},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824037,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48128,"kb_used_data":15540,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94323712,"statfs":{"total":96636764160,"available":96587481088,"internally_reserved":0,"allocated":15912960,"data_stored":13661725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509593,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41472,"kb_used_data":10284,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330368,"statfs":{"total":96636764160,"available":96594296832,"internally_reserved":0,"allocated":10530816,"data_stored":8293183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509593,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48648,"kb_used_data":17460,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323192,"statfs":{"total":96636764160,"available":96586948608,"internally_reserved":0,"allocated":17879040,"data_stored":15641407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":298,"seq":1279900254224,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47516,"kb_used_data":14028,"kb_used_omap":10,"kb_used_meta":33461,"kb_avail":94324324,"statfs":{"total":96636764160,"available":96588107776,"internally_reserved":0,"allocated":14364672,"data_stored":12113437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10986,"internal_metadata":34264342},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575002,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47668,"kb_used_data":16484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324172,"statfs":{"total":96636764160,"available":96587952128,"internally_reserved":0,"allocated":16879616,"data_stored":14628381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:08:03.675 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:08:28.677 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:08:28.677 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:08:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:28.809+0000 7f3b6f2d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:28.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:28.821+0000 7f3b6f2d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:28.995 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:08:28.995 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:08:29.014 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":724,"stamp":"2024-03-30T13:08:28.004399+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":297933630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":346860,"kb_used_data":122960,"kb_used_omap":34,"kb_used_meta":223773,"kb_avail":660256020,"statfs":{"total":676457349120,"available":676102164480,"internally_reserved":0,"allocated":125911040,"data_stored":109821575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35570,"internal_metadata":229143822},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.612493"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":781,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-03-30T13:08:18.019001+0000","last_change":"2024-03-30T13:07:45.158080+0000","last_active":"2024-03-30T13:08:18.019001+0000","last_peered":"2024-03-30T13:08:18.019001+0000","last_clean":"2024-03-30T13:08:18.019001+0000","last_became_active":"2024-03-30T13:07:45.156785+0000","last_became_peered":"2024-03-30T13:07:45.156785+0000","last_unstale":"2024-03-30T13:08:18.019001+0000","last_undegraded":"2024-03-30T13:08:18.019001+0000","last_fullsized":"2024-03-30T13:08:18.019001+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:38.717555+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:07:38.717555+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:57.207615+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22941225,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":723,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-03-30T13:08:17.867195+0000","last_change":"2024-03-30T13:07:26.626194+0000","last_active":"2024-03-30T13:08:17.867195+0000","last_peered":"2024-03-30T13:08:17.867195+0000","last_clean":"2024-03-30T13:08:17.867195+0000","last_became_active":"2024-03-30T13:06:54.153354+0000","last_became_peered":"2024-03-30T13:06:54.153354+0000","last_unstale":"2024-03-30T13:08:17.867195+0000","last_undegraded":"2024-03-30T13:08:17.867195+0000","last_fullsized":"2024-03-30T13:08:17.867195+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:07:26.625854+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:07:26.625854+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:30.881671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":339,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-03-30T13:08:16.394332+0000","last_change":"2024-03-30T13:08:04.426459+0000","last_active":"2024-03-30T13:08:16.394332+0000","last_peered":"2024-03-30T13:08:16.394332+0000","last_clean":"2024-03-30T13:08:16.394332+0000","last_became_active":"2024-03-30T13:08:04.348827+0000","last_became_peered":"2024-03-30T13:08:04.348827+0000","last_unstale":"2024-03-30T13:08:16.394332+0000","last_undegraded":"2024-03-30T13:08:16.394332+0000","last_fullsized":"2024-03-30T13:08:16.394332+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:07:23.683745+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:03:39.460563+0000","last_clean_scrub_stamp":"2024-03-30T13:07:23.683745+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:45.754231+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":312,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-03-30T13:08:16.394261+0000","last_change":"2024-03-30T13:07:16.522990+0000","last_active":"2024-03-30T13:08:16.394261+0000","last_peered":"2024-03-30T13:08:16.394261+0000","last_clean":"2024-03-30T13:08:16.394261+0000","last_became_active":"2024-03-30T13:06:54.156927+0000","last_became_peered":"2024-03-30T13:06:54.156927+0000","last_unstale":"2024-03-30T13:08:16.394261+0000","last_undegraded":"2024-03-30T13:08:16.394261+0000","last_fullsized":"2024-03-30T13:08:16.394261+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:16.522745+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:07:16.522745+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:40.030402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":109,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-03-30T13:08:17.400010+0000","last_change":"2024-03-30T13:07:45.158822+0000","last_active":"2024-03-30T13:08:17.400010+0000","last_peered":"2024-03-30T13:08:17.400010+0000","last_clean":"2024-03-30T13:08:17.400010+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:08:17.400010+0000","last_undegraded":"2024-03-30T13:08:17.400010+0000","last_fullsized":"2024-03-30T13:08:17.400010+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:07:37.576416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:07:37.576416+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:50.550338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":823,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-03-30T13:08:17.513727+0000","last_change":"2024-03-30T13:07:45.157963+0000","last_active":"2024-03-30T13:08:17.513727+0000","last_peered":"2024-03-30T13:08:17.513727+0000","last_clean":"2024-03-30T13:08:17.513727+0000","last_became_active":"2024-03-30T13:07:45.156174+0000","last_became_peered":"2024-03-30T13:07:45.156174+0000","last_unstale":"2024-03-30T13:08:17.513727+0000","last_undegraded":"2024-03-30T13:08:17.513727+0000","last_fullsized":"2024-03-30T13:08:17.513727+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_clean_scrub_stamp":"2024-03-30T13:07:40.613260+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:54.625356+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":56974905,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":339,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-03-30T13:08:16.394339+0000","last_change":"2024-03-30T13:07:08.403925+0000","last_active":"2024-03-30T13:08:16.394339+0000","last_peered":"2024-03-30T13:08:16.394339+0000","last_clean":"2024-03-30T13:08:16.394339+0000","last_became_active":"2024-03-30T13:06:54.155351+0000","last_became_peered":"2024-03-30T13:06:54.155351+0000","last_unstale":"2024-03-30T13:08:16.394339+0000","last_undegraded":"2024-03-30T13:08:16.394339+0000","last_fullsized":"2024-03-30T13:08:16.394339+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:07:08.403663+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:07:08.403663+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:24.852917+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":27,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-03-30T13:08:17.867014+0000","last_change":"2024-03-30T13:08:04.407774+0000","last_active":"2024-03-30T13:08:17.867014+0000","last_peered":"2024-03-30T13:08:17.867014+0000","last_clean":"2024-03-30T13:08:17.867014+0000","last_became_active":"2024-03-30T13:08:04.348288+0000","last_became_peered":"2024-03-30T13:08:04.348288+0000","last_unstale":"2024-03-30T13:08:17.867014+0000","last_undegraded":"2024-03-30T13:08:17.867014+0000","last_fullsized":"2024-03-30T13:08:17.867014+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:07:44.108131+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:07:44.108131+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:47.119015+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":894,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-03-30T13:08:17.513860+0000","last_change":"2024-03-30T13:07:45.156994+0000","last_active":"2024-03-30T13:08:17.513860+0000","last_peered":"2024-03-30T13:08:17.513860+0000","last_clean":"2024-03-30T13:08:17.513860+0000","last_became_active":"2024-03-30T13:07:45.156128+0000","last_became_peered":"2024-03-30T13:07:45.156128+0000","last_unstale":"2024-03-30T13:08:17.513860+0000","last_undegraded":"2024-03-30T13:08:17.513860+0000","last_fullsized":"2024-03-30T13:08:17.513860+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:05.491728+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:40.339270+0000","last_clean_scrub_stamp":"2024-03-30T13:07:05.491728+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:28.374332+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":65983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":633,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-03-30T13:08:17.867063+0000","last_change":"2024-03-30T13:08:04.573877+0000","last_active":"2024-03-30T13:08:17.867063+0000","last_peered":"2024-03-30T13:08:17.867063+0000","last_clean":"2024-03-30T13:08:17.867063+0000","last_became_active":"2024-03-30T13:05:07.445793+0000","last_became_peered":"2024-03-30T13:05:07.445793+0000","last_unstale":"2024-03-30T13:08:17.867063+0000","last_undegraded":"2024-03-30T13:08:17.867063+0000","last_fullsized":"2024-03-30T13:08:17.867063+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:08:04.573631+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:08:04.573631+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:27.130867+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":786,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-03-30T13:08:18.018940+0000","last_change":"2024-03-30T13:07:48.693563+0000","last_active":"2024-03-30T13:08:18.018940+0000","last_peered":"2024-03-30T13:08:18.018940+0000","last_clean":"2024-03-30T13:08:18.018940+0000","last_became_active":"2024-03-30T13:05:07.445475+0000","last_became_peered":"2024-03-30T13:05:07.445475+0000","last_unstale":"2024-03-30T13:08:18.018940+0000","last_undegraded":"2024-03-30T13:08:18.018940+0000","last_fullsized":"2024-03-30T13:08:18.018940+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:07:48.693155+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:49.070756+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":835,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-03-30T13:08:18.018991+0000","last_change":"2024-03-30T13:07:45.158029+0000","last_active":"2024-03-30T13:08:18.018991+0000","last_peered":"2024-03-30T13:08:18.018991+0000","last_clean":"2024-03-30T13:08:18.018991+0000","last_became_active":"2024-03-30T13:07:45.156745+0000","last_became_peered":"2024-03-30T13:07:45.156745+0000","last_unstale":"2024-03-30T13:08:18.018991+0000","last_undegraded":"2024-03-30T13:08:18.018991+0000","last_fullsized":"2024-03-30T13:08:18.018991+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:07:24.693213+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:07:24.693213+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:38.979624+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":57731253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":125,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-03-30T13:08:17.400007+0000","last_change":"2024-03-30T13:07:53.686045+0000","last_active":"2024-03-30T13:08:17.400007+0000","last_peered":"2024-03-30T13:08:17.400007+0000","last_clean":"2024-03-30T13:08:17.400007+0000","last_became_active":"2024-03-30T13:07:45.154855+0000","last_became_peered":"2024-03-30T13:07:45.154855+0000","last_unstale":"2024-03-30T13:08:17.400007+0000","last_undegraded":"2024-03-30T13:08:17.400007+0000","last_fullsized":"2024-03-30T13:08:17.400007+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:07:53.685778+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:58.028417+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":277,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-03-30T13:08:16.394270+0000","last_change":"2024-03-30T13:07:01.439676+0000","last_active":"2024-03-30T13:08:16.394270+0000","last_peered":"2024-03-30T13:08:16.394270+0000","last_clean":"2024-03-30T13:08:16.394270+0000","last_became_active":"2024-03-30T13:06:54.153682+0000","last_became_peered":"2024-03-30T13:06:54.153682+0000","last_unstale":"2024-03-30T13:08:16.394270+0000","last_undegraded":"2024-03-30T13:08:16.394270+0000","last_fullsized":"2024-03-30T13:08:16.394270+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:07:01.439296+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:07:01.439296+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:21.420094+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":133,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-03-30T13:08:17.400002+0000","last_change":"2024-03-30T13:08:07.461164+0000","last_active":"2024-03-30T13:08:17.400002+0000","last_peered":"2024-03-30T13:08:17.400002+0000","last_clean":"2024-03-30T13:08:17.400002+0000","last_became_active":"2024-03-30T13:07:45.154381+0000","last_became_peered":"2024-03-30T13:07:45.154381+0000","last_unstale":"2024-03-30T13:08:17.400002+0000","last_undegraded":"2024-03-30T13:08:17.400002+0000","last_fullsized":"2024-03-30T13:08:17.400002+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:08:07.460876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:08:07.460876+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:20.973897+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":25,"reported_epoch":343,"state":"active+clean","last_fresh":"2024-03-30T13:08:16.397509+0000","last_change":"2024-03-30T13:08:04.399255+0000","last_active":"2024-03-30T13:08:16.397509+0000","last_peered":"2024-03-30T13:08:16.397509+0000","last_clean":"2024-03-30T13:08:16.397509+0000","last_became_active":"2024-03-30T13:08:04.352075+0000","last_became_peered":"2024-03-30T13:08:04.352075+0000","last_unstale":"2024-03-30T13:08:16.397509+0000","last_undegraded":"2024-03-30T13:08:16.397509+0000","last_fullsized":"2024-03-30T13:08:16.397509+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:38.221640+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:07:38.221640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:02.186618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5226911,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":297933630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102043648,"data_stored":102043648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":227,"seq":974957576245,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50868,"kb_used_data":19176,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94320972,"statfs":{"total":96636764160,"available":96584675328,"internally_reserved":0,"allocated":19636224,"data_stored":17337729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534730,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56512,"kb_used_data":23920,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94315328,"statfs":{"total":96636764160,"available":96578895872,"internally_reserved":0,"allocated":24494080,"data_stored":22195585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":33350120},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824042,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50164,"kb_used_data":17576,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94321676,"statfs":{"total":96636764160,"available":96585396224,"internally_reserved":0,"allocated":17997824,"data_stored":15699329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509598,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45224,"kb_used_data":14040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326616,"statfs":{"total":96636764160,"available":96590454784,"internally_reserved":0,"allocated":14376960,"data_stored":12078465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509598,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48748,"kb_used_data":17568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323092,"statfs":{"total":96636764160,"available":96586846208,"internally_reserved":0,"allocated":17989632,"data_stored":15691137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":298,"seq":1279900254228,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47596,"kb_used_data":14112,"kb_used_omap":10,"kb_used_meta":33461,"kb_avail":94324244,"statfs":{"total":96636764160,"available":96588025856,"internally_reserved":0,"allocated":14450688,"data_stored":12152193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10986,"internal_metadata":34264342},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575007,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47748,"kb_used_data":16568,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324092,"statfs":{"total":96636764160,"available":96587870208,"internally_reserved":0,"allocated":16965632,"data_stored":14667137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292032,"data_stored":5292032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10211328,"data_stored":10211328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":13832192,"data_stored":13832192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:08:29.015 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:08:29.015 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:08:29.015 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-03-30T13:08:29.015 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-30T13:08:29.015 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-30T13:08:29.017 INFO:tasks.ceph.osd.1:Started 2024-03-30T13:08:29.018 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-30T13:08:29.077 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:08:29.077+0000 7f9c1df117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:29.087 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:08:29.089+0000 7f9c1df117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:29.088 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:08:29.089+0000 7f9c1df117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:29.131 INFO:teuthology.orchestra.run.smithi067.stderr:no valid command found; 10 closest matches: 2024-03-30T13:08:29.131 INFO:teuthology.orchestra.run.smithi067.stderr:0 2024-03-30T13:08:29.131 INFO:teuthology.orchestra.run.smithi067.stderr:1 2024-03-30T13:08:29.132 INFO:teuthology.orchestra.run.smithi067.stderr:2 2024-03-30T13:08:29.132 INFO:teuthology.orchestra.run.smithi067.stderr:abort 2024-03-30T13:08:29.132 INFO:teuthology.orchestra.run.smithi067.stderr:assert 2024-03-30T13:08:29.132 INFO:teuthology.orchestra.run.smithi067.stderr:config diff 2024-03-30T13:08:29.132 INFO:teuthology.orchestra.run.smithi067.stderr:config diff get 2024-03-30T13:08:29.132 INFO:teuthology.orchestra.run.smithi067.stderr:config get 2024-03-30T13:08:29.132 INFO:teuthology.orchestra.run.smithi067.stderr:config help [] 2024-03-30T13:08:29.132 INFO:teuthology.orchestra.run.smithi067.stderr:config set ... 2024-03-30T13:08:29.132 INFO:teuthology.orchestra.run.smithi067.stderr:admin_socket: invalid command 2024-03-30T13:08:29.134 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:08:29.134 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-30T13:08:29.619 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:08:29.621+0000 7f9c1df117c0 -1 Falling back to public interface 2024-03-30T13:08:33.360 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:08:33.361+0000 7f9c1df117c0 -1 osd.1 336 log_to_monitors true 2024-03-30T13:08:33.494 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:08:33.497+0000 7f9c15ce6640 -1 osd.1 336 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:08:34.135 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-30T13:08:34.271 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-30T13:08:34.271 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-30T13:08:34.271 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-03-30T13:08:34.271 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-30T13:08:34.286 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:08:34.286 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-03-30T13:08:34.287 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 1 2024-03-30T13:08:34.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:34.401+0000 7fb0577b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:34.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:34.417+0000 7fb0577b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:34.604 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:08:34.605+0000 7fbb38f08640 -1 mon.a@0(leader).osd e346 definitely_dead 0 2024-03-30T13:08:35.491 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.1. 2024-03-30T13:08:35.521 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-03-30T13:08:35.521 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:08:35.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:35.637+0000 7f6d17274640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:35.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:35.657+0000 7f6d17274640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:36.494 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:08:36.524 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:08:36.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:36.637+0000 7f3f40442640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:36.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:36.649+0000 7f3f40442640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:37.503 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:08:37.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:08:37.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:37.645+0000 7f75d961e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:37.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:37.661+0000 7f75d961e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:38.526 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:08:38.555 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:08:38.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:38.677+0000 7f2ebf896640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:38.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:38.693+0000 7f2ebf896640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:39.529 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:08:39.556 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:08:39.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:39.669+0000 7f10196c6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:39.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:39.685+0000 7f10196c6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:40.541 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:08:40.566 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:08:40.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:40.685+0000 7f304f87d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:40.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:40.705+0000 7f304f87d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:41.559 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:08:41.582 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:08:41.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:41.705+0000 7f3243f58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:41.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:41.721+0000 7f3243f58640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:42.571 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:08:42.594 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:08:42.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:42.713+0000 7fb1c5689640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:42.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:42.725+0000 7fb1c5689640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:42.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:42.933+0000 7fb1beffd640 0 --2- 172.21.15.67:0/3593107931 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb1c0066d30 0x7fb1c00aaf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-03-30T13:08:43.580 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:08:43.606 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:08:43.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:43.725+0000 7efee2262640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:43.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:43.745+0000 7efee2262640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:44.587 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:08:44.615 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:08:44.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:44.737+0000 7f4526e4b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:44.753+0000 7f4526e4b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:45.604 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:08:45.630 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:08:45.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:45.753+0000 7fc016432640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:45.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:45.773+0000 7fc016432640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:46.620 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:08:46.644 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:08:46.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:46.765+0000 7fde032ad640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:46.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:46.781+0000 7fde032ad640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:47.623 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:08:47.649 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:08:47.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:47.765+0000 7f95689e8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:47.781+0000 7f95689e8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:48.633 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:08:48.656 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:08:48.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:48.781+0000 7f00cdc1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:48.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:48.797+0000 7f00cdc1d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:49.636 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:08:49.661 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:08:49.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:49.781+0000 7fda9ff99640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:49.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:49.797+0000 7fda9ff99640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:50.649 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:08:50.678 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:08:50.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:50.801+0000 7f7ec21e8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:50.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:50.817+0000 7f7ec21e8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:51.673 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:08:51.696 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:08:51.696 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:08:51.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:51.821+0000 7f018f781640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:51.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:51.833+0000 7f018f781640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:52.006 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:08:52.006 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:08:52.024 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":755,"stamp":"2024-03-30T13:08:51.671517+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":300521875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":49,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":384636,"kb_used_data":128588,"kb_used_omap":40,"kb_used_meta":255895,"kb_avail":754590084,"statfs":{"total":773094113280,"available":772700246016,"internally_reserved":0,"allocated":131674112,"data_stored":112420464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41615,"internal_metadata":262036849},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.051106"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":816,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-30T13:08:49.641841+0000","last_change":"2024-03-30T13:07:45.158080+0000","last_active":"2024-03-30T13:08:49.641841+0000","last_peered":"2024-03-30T13:08:49.641841+0000","last_clean":"2024-03-30T13:08:49.641841+0000","last_became_active":"2024-03-30T13:07:45.156785+0000","last_became_peered":"2024-03-30T13:07:45.156785+0000","last_unstale":"2024-03-30T13:08:49.641841+0000","last_undegraded":"2024-03-30T13:08:49.641841+0000","last_fullsized":"2024-03-30T13:08:49.641841+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:38.717555+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:07:38.717555+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:57.207615+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22941225,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":767,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-30T13:08:50.123856+0000","last_change":"2024-03-30T13:08:32.414902+0000","last_active":"2024-03-30T13:08:50.123856+0000","last_peered":"2024-03-30T13:08:50.123856+0000","last_clean":"2024-03-30T13:08:50.123856+0000","last_became_active":"2024-03-30T13:06:54.153354+0000","last_became_peered":"2024-03-30T13:06:54.153354+0000","last_unstale":"2024-03-30T13:08:50.123856+0000","last_undegraded":"2024-03-30T13:08:50.123856+0000","last_fullsized":"2024-03-30T13:08:50.123856+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:08:32.414658+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:08:32.414658+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:38.583836+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":400,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-03-30T13:08:47.757223+0000","last_change":"2024-03-30T13:08:39.542911+0000","last_active":"2024-03-30T13:08:47.757223+0000","last_peered":"2024-03-30T13:08:47.757223+0000","last_clean":"2024-03-30T13:08:47.757223+0000","last_became_active":"2024-03-30T13:08:39.541896+0000","last_became_peered":"2024-03-30T13:08:39.541896+0000","last_unstale":"2024-03-30T13:08:47.757223+0000","last_undegraded":"2024-03-30T13:08:47.757223+0000","last_fullsized":"2024-03-30T13:08:47.757223+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:08:37.747009+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:43.320096+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":352,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-03-30T13:08:47.761786+0000","last_change":"2024-03-30T13:08:47.761868+0000","last_active":"2024-03-30T13:08:47.761786+0000","last_peered":"2024-03-30T13:08:47.761786+0000","last_clean":"2024-03-30T13:08:47.761786+0000","last_became_active":"2024-03-30T13:06:54.156927+0000","last_became_peered":"2024-03-30T13:06:54.156927+0000","last_unstale":"2024-03-30T13:08:47.761786+0000","last_undegraded":"2024-03-30T13:08:47.761786+0000","last_fullsized":"2024-03-30T13:08:47.761786+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:08:47.761560+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:08:47.761560+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:53.763939+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":139,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-03-30T13:08:48.211091+0000","last_change":"2024-03-30T13:07:45.158822+0000","last_active":"2024-03-30T13:08:48.211091+0000","last_peered":"2024-03-30T13:08:48.211091+0000","last_clean":"2024-03-30T13:08:48.211091+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:08:48.211091+0000","last_undegraded":"2024-03-30T13:08:48.211091+0000","last_fullsized":"2024-03-30T13:08:48.211091+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:07:37.576416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:07:37.576416+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:50.550338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":851,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-03-30T13:08:47.624949+0000","last_change":"2024-03-30T13:07:45.157963+0000","last_active":"2024-03-30T13:08:47.624949+0000","last_peered":"2024-03-30T13:08:47.624949+0000","last_clean":"2024-03-30T13:08:47.624949+0000","last_became_active":"2024-03-30T13:07:45.156174+0000","last_became_peered":"2024-03-30T13:07:45.156174+0000","last_unstale":"2024-03-30T13:08:47.624949+0000","last_undegraded":"2024-03-30T13:08:47.624949+0000","last_fullsized":"2024-03-30T13:08:47.624949+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_clean_scrub_stamp":"2024-03-30T13:07:40.613260+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:54.625356+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":56974905,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":379,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-03-30T13:08:47.757115+0000","last_change":"2024-03-30T13:08:26.697182+0000","last_active":"2024-03-30T13:08:47.757115+0000","last_peered":"2024-03-30T13:08:47.757115+0000","last_clean":"2024-03-30T13:08:47.757115+0000","last_became_active":"2024-03-30T13:06:54.155351+0000","last_became_peered":"2024-03-30T13:06:54.155351+0000","last_unstale":"2024-03-30T13:08:47.757115+0000","last_undegraded":"2024-03-30T13:08:47.757115+0000","last_fullsized":"2024-03-30T13:08:47.757115+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:08:26.696919+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:08:26.696919+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:38.403041+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":69,"reported_epoch":361,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-30T13:08:50.123472+0000","last_change":"2024-03-30T13:08:37.516659+0000","last_active":"2024-03-30T13:08:50.123472+0000","last_peered":"2024-03-30T13:08:50.123472+0000","last_clean":"2024-03-30T13:08:34.571093+0000","last_became_active":"2024-03-30T13:08:37.514196+0000","last_became_peered":"2024-03-30T13:08:37.514196+0000","last_unstale":"2024-03-30T13:08:50.123472+0000","last_undegraded":"2024-03-30T13:08:50.123472+0000","last_fullsized":"2024-03-30T13:08:50.123472+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:07:44.108131+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:07:44.108131+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[4,6,5],"avail_no_missing":["4(0)","1(0)","5(2)","6(1)"],"object_location_counts":[{"shards":"4(0),5(2),6(1)","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":930,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-03-30T13:08:47.625172+0000","last_change":"2024-03-30T13:08:29.496007+0000","last_active":"2024-03-30T13:08:47.625172+0000","last_peered":"2024-03-30T13:08:47.625172+0000","last_clean":"2024-03-30T13:08:47.625172+0000","last_became_active":"2024-03-30T13:07:45.156128+0000","last_became_peered":"2024-03-30T13:07:45.156128+0000","last_unstale":"2024-03-30T13:08:47.625172+0000","last_undegraded":"2024-03-30T13:08:47.625172+0000","last_fullsized":"2024-03-30T13:08:47.625172+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:08:29.495729+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:08:29.495729+0000","last_clean_scrub_stamp":"2024-03-30T13:08:29.495729+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:32.078835+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":65983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":669,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-30T13:08:50.123472+0000","last_change":"2024-03-30T13:08:04.573877+0000","last_active":"2024-03-30T13:08:50.123472+0000","last_peered":"2024-03-30T13:08:50.123472+0000","last_clean":"2024-03-30T13:08:50.123472+0000","last_became_active":"2024-03-30T13:05:07.445793+0000","last_became_peered":"2024-03-30T13:05:07.445793+0000","last_unstale":"2024-03-30T13:08:50.123472+0000","last_undegraded":"2024-03-30T13:08:50.123472+0000","last_fullsized":"2024-03-30T13:08:50.123472+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:08:04.573631+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:08:04.573631+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:27.130867+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":829,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-30T13:08:49.641674+0000","last_change":"2024-03-30T13:08:49.546393+0000","last_active":"2024-03-30T13:08:49.641674+0000","last_peered":"2024-03-30T13:08:49.641674+0000","last_clean":"2024-03-30T13:08:49.641674+0000","last_became_active":"2024-03-30T13:05:07.445475+0000","last_became_peered":"2024-03-30T13:05:07.445475+0000","last_unstale":"2024-03-30T13:08:49.641674+0000","last_undegraded":"2024-03-30T13:08:49.641674+0000","last_fullsized":"2024-03-30T13:08:49.641674+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:08:49.546103+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:08:49.546103+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:50.816946+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":878,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-30T13:08:49.641749+0000","last_change":"2024-03-30T13:08:41.532360+0000","last_active":"2024-03-30T13:08:49.641749+0000","last_peered":"2024-03-30T13:08:49.641749+0000","last_clean":"2024-03-30T13:08:49.641749+0000","last_became_active":"2024-03-30T13:07:45.156745+0000","last_became_peered":"2024-03-30T13:07:45.156745+0000","last_unstale":"2024-03-30T13:08:49.641749+0000","last_undegraded":"2024-03-30T13:08:49.641749+0000","last_fullsized":"2024-03-30T13:08:49.641749+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:08:41.532076+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:08:41.532076+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:43.071429+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":57731253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":155,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-03-30T13:08:48.211082+0000","last_change":"2024-03-30T13:07:53.686045+0000","last_active":"2024-03-30T13:08:48.211082+0000","last_peered":"2024-03-30T13:08:48.211082+0000","last_clean":"2024-03-30T13:08:48.211082+0000","last_became_active":"2024-03-30T13:07:45.154855+0000","last_became_peered":"2024-03-30T13:07:45.154855+0000","last_unstale":"2024-03-30T13:08:48.211082+0000","last_undegraded":"2024-03-30T13:08:48.211082+0000","last_fullsized":"2024-03-30T13:08:48.211082+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:07:53.685778+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:58.028417+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":317,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-03-30T13:08:47.757102+0000","last_change":"2024-03-30T13:08:24.700616+0000","last_active":"2024-03-30T13:08:47.757102+0000","last_peered":"2024-03-30T13:08:47.757102+0000","last_clean":"2024-03-30T13:08:47.757102+0000","last_became_active":"2024-03-30T13:06:54.153682+0000","last_became_peered":"2024-03-30T13:06:54.153682+0000","last_unstale":"2024-03-30T13:08:47.757102+0000","last_undegraded":"2024-03-30T13:08:47.757102+0000","last_fullsized":"2024-03-30T13:08:47.757102+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:08:24.700361+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:08:24.700361+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:54.523068+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":163,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-03-30T13:08:48.211120+0000","last_change":"2024-03-30T13:08:07.461164+0000","last_active":"2024-03-30T13:08:48.211120+0000","last_peered":"2024-03-30T13:08:48.211120+0000","last_clean":"2024-03-30T13:08:48.211120+0000","last_became_active":"2024-03-30T13:07:45.154381+0000","last_became_peered":"2024-03-30T13:07:45.154381+0000","last_unstale":"2024-03-30T13:08:48.211120+0000","last_undegraded":"2024-03-30T13:08:48.211120+0000","last_fullsized":"2024-03-30T13:08:48.211120+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:08:07.460876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:08:07.460876+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:20.973897+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":75,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-03-30T13:08:48.640675+0000","last_change":"2024-03-30T13:08:39.544104+0000","last_active":"2024-03-30T13:08:48.640675+0000","last_peered":"2024-03-30T13:08:48.640675+0000","last_clean":"2024-03-30T13:08:48.640675+0000","last_became_active":"2024-03-30T13:08:39.543465+0000","last_became_peered":"2024-03-30T13:08:39.543465+0000","last_unstale":"2024-03-30T13:08:48.640675+0000","last_undegraded":"2024-03-30T13:08:48.640675+0000","last_fullsized":"2024-03-30T13:08:48.640675+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:38.221640+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:07:38.221640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":300521875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":227,"seq":974957576250,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51060,"kb_used_data":19368,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94320780,"statfs":{"total":96636764160,"available":96584478720,"internally_reserved":0,"allocated":19832832,"data_stored":17427818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534735,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56708,"kb_used_data":24112,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94315132,"statfs":{"total":96636764160,"available":96578695168,"internally_reserved":0,"allocated":24690688,"data_stored":22285674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":33350120},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824047,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50384,"kb_used_data":17792,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94321456,"statfs":{"total":96636764160,"available":96585170944,"internally_reserved":0,"allocated":18219008,"data_stored":15800458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509603,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41780,"kb_used_data":10596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330060,"statfs":{"total":96636764160,"available":96593981440,"internally_reserved":0,"allocated":10850304,"data_stored":8438522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":346,"seq":1486058684421,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40060,"kb_used_data":7920,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94331780,"statfs":{"total":96636764160,"available":96595742720,"internally_reserved":0,"allocated":8110080,"data_stored":5698298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6045,"internal_metadata":32893027},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509603,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48940,"kb_used_data":17760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322900,"statfs":{"total":96636764160,"available":96586649600,"internally_reserved":0,"allocated":18186240,"data_stored":15781226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":298,"seq":1279900254233,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47776,"kb_used_data":14292,"kb_used_omap":10,"kb_used_meta":33461,"kb_avail":94324064,"statfs":{"total":96636764160,"available":96587841536,"internally_reserved":0,"allocated":14635008,"data_stored":12236762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10986,"internal_metadata":34264342},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575011,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47928,"kb_used_data":16748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323912,"statfs":{"total":96636764160,"available":96587685888,"internally_reserved":0,"allocated":17149952,"data_stored":14751706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3735552,"data_stored":3735552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":13832192,"data_stored":13832192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:08:52.026 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:08:52.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:52.145+0000 7fe9d89a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:52.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:52.157+0000 7fe9d89a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:52.336 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:08:52.336 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:08:52.354 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":756,"stamp":"2024-03-30T13:08:52.011175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":300521875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":49,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":384636,"kb_used_data":128588,"kb_used_omap":40,"kb_used_meta":255895,"kb_avail":754590084,"statfs":{"total":773094113280,"available":772700246016,"internally_reserved":0,"allocated":131674112,"data_stored":112420464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41615,"internal_metadata":262036849},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001164"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":816,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-30T13:08:49.641841+0000","last_change":"2024-03-30T13:07:45.158080+0000","last_active":"2024-03-30T13:08:49.641841+0000","last_peered":"2024-03-30T13:08:49.641841+0000","last_clean":"2024-03-30T13:08:49.641841+0000","last_became_active":"2024-03-30T13:07:45.156785+0000","last_became_peered":"2024-03-30T13:07:45.156785+0000","last_unstale":"2024-03-30T13:08:49.641841+0000","last_undegraded":"2024-03-30T13:08:49.641841+0000","last_fullsized":"2024-03-30T13:08:49.641841+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:38.717555+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:07:38.717555+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:57.207615+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22941225,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":767,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-30T13:08:50.123856+0000","last_change":"2024-03-30T13:08:32.414902+0000","last_active":"2024-03-30T13:08:50.123856+0000","last_peered":"2024-03-30T13:08:50.123856+0000","last_clean":"2024-03-30T13:08:50.123856+0000","last_became_active":"2024-03-30T13:06:54.153354+0000","last_became_peered":"2024-03-30T13:06:54.153354+0000","last_unstale":"2024-03-30T13:08:50.123856+0000","last_undegraded":"2024-03-30T13:08:50.123856+0000","last_fullsized":"2024-03-30T13:08:50.123856+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:08:32.414658+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:08:32.414658+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:38.583836+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":400,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-03-30T13:08:47.757223+0000","last_change":"2024-03-30T13:08:39.542911+0000","last_active":"2024-03-30T13:08:47.757223+0000","last_peered":"2024-03-30T13:08:47.757223+0000","last_clean":"2024-03-30T13:08:47.757223+0000","last_became_active":"2024-03-30T13:08:39.541896+0000","last_became_peered":"2024-03-30T13:08:39.541896+0000","last_unstale":"2024-03-30T13:08:47.757223+0000","last_undegraded":"2024-03-30T13:08:47.757223+0000","last_fullsized":"2024-03-30T13:08:47.757223+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:08:37.747009+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:43.320096+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":352,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-03-30T13:08:47.761786+0000","last_change":"2024-03-30T13:08:47.761868+0000","last_active":"2024-03-30T13:08:47.761786+0000","last_peered":"2024-03-30T13:08:47.761786+0000","last_clean":"2024-03-30T13:08:47.761786+0000","last_became_active":"2024-03-30T13:06:54.156927+0000","last_became_peered":"2024-03-30T13:06:54.156927+0000","last_unstale":"2024-03-30T13:08:47.761786+0000","last_undegraded":"2024-03-30T13:08:47.761786+0000","last_fullsized":"2024-03-30T13:08:47.761786+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:08:47.761560+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:08:47.761560+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:53.763939+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":139,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-03-30T13:08:48.211091+0000","last_change":"2024-03-30T13:07:45.158822+0000","last_active":"2024-03-30T13:08:48.211091+0000","last_peered":"2024-03-30T13:08:48.211091+0000","last_clean":"2024-03-30T13:08:48.211091+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:08:48.211091+0000","last_undegraded":"2024-03-30T13:08:48.211091+0000","last_fullsized":"2024-03-30T13:08:48.211091+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:07:37.576416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:07:37.576416+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:50.550338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":851,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-03-30T13:08:47.624949+0000","last_change":"2024-03-30T13:07:45.157963+0000","last_active":"2024-03-30T13:08:47.624949+0000","last_peered":"2024-03-30T13:08:47.624949+0000","last_clean":"2024-03-30T13:08:47.624949+0000","last_became_active":"2024-03-30T13:07:45.156174+0000","last_became_peered":"2024-03-30T13:07:45.156174+0000","last_unstale":"2024-03-30T13:08:47.624949+0000","last_undegraded":"2024-03-30T13:08:47.624949+0000","last_fullsized":"2024-03-30T13:08:47.624949+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_clean_scrub_stamp":"2024-03-30T13:07:40.613260+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:54.625356+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":56974905,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":379,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-03-30T13:08:47.757115+0000","last_change":"2024-03-30T13:08:26.697182+0000","last_active":"2024-03-30T13:08:47.757115+0000","last_peered":"2024-03-30T13:08:47.757115+0000","last_clean":"2024-03-30T13:08:47.757115+0000","last_became_active":"2024-03-30T13:06:54.155351+0000","last_became_peered":"2024-03-30T13:06:54.155351+0000","last_unstale":"2024-03-30T13:08:47.757115+0000","last_undegraded":"2024-03-30T13:08:47.757115+0000","last_fullsized":"2024-03-30T13:08:47.757115+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:08:26.696919+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:08:26.696919+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:38.403041+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":69,"reported_epoch":361,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-30T13:08:50.123472+0000","last_change":"2024-03-30T13:08:37.516659+0000","last_active":"2024-03-30T13:08:50.123472+0000","last_peered":"2024-03-30T13:08:50.123472+0000","last_clean":"2024-03-30T13:08:34.571093+0000","last_became_active":"2024-03-30T13:08:37.514196+0000","last_became_peered":"2024-03-30T13:08:37.514196+0000","last_unstale":"2024-03-30T13:08:50.123472+0000","last_undegraded":"2024-03-30T13:08:50.123472+0000","last_fullsized":"2024-03-30T13:08:50.123472+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:07:44.108131+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:07:44.108131+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[4,6,5],"avail_no_missing":["4(0)","1(0)","5(2)","6(1)"],"object_location_counts":[{"shards":"4(0),5(2),6(1)","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":930,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-03-30T13:08:47.625172+0000","last_change":"2024-03-30T13:08:29.496007+0000","last_active":"2024-03-30T13:08:47.625172+0000","last_peered":"2024-03-30T13:08:47.625172+0000","last_clean":"2024-03-30T13:08:47.625172+0000","last_became_active":"2024-03-30T13:07:45.156128+0000","last_became_peered":"2024-03-30T13:07:45.156128+0000","last_unstale":"2024-03-30T13:08:47.625172+0000","last_undegraded":"2024-03-30T13:08:47.625172+0000","last_fullsized":"2024-03-30T13:08:47.625172+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:08:29.495729+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:08:29.495729+0000","last_clean_scrub_stamp":"2024-03-30T13:08:29.495729+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:32.078835+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":65983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":669,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-30T13:08:50.123472+0000","last_change":"2024-03-30T13:08:04.573877+0000","last_active":"2024-03-30T13:08:50.123472+0000","last_peered":"2024-03-30T13:08:50.123472+0000","last_clean":"2024-03-30T13:08:50.123472+0000","last_became_active":"2024-03-30T13:05:07.445793+0000","last_became_peered":"2024-03-30T13:05:07.445793+0000","last_unstale":"2024-03-30T13:08:50.123472+0000","last_undegraded":"2024-03-30T13:08:50.123472+0000","last_fullsized":"2024-03-30T13:08:50.123472+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:08:04.573631+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:08:04.573631+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:27.130867+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":829,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-30T13:08:49.641674+0000","last_change":"2024-03-30T13:08:49.546393+0000","last_active":"2024-03-30T13:08:49.641674+0000","last_peered":"2024-03-30T13:08:49.641674+0000","last_clean":"2024-03-30T13:08:49.641674+0000","last_became_active":"2024-03-30T13:05:07.445475+0000","last_became_peered":"2024-03-30T13:05:07.445475+0000","last_unstale":"2024-03-30T13:08:49.641674+0000","last_undegraded":"2024-03-30T13:08:49.641674+0000","last_fullsized":"2024-03-30T13:08:49.641674+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:08:49.546103+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:08:49.546103+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:50.816946+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":878,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-30T13:08:49.641749+0000","last_change":"2024-03-30T13:08:41.532360+0000","last_active":"2024-03-30T13:08:49.641749+0000","last_peered":"2024-03-30T13:08:49.641749+0000","last_clean":"2024-03-30T13:08:49.641749+0000","last_became_active":"2024-03-30T13:07:45.156745+0000","last_became_peered":"2024-03-30T13:07:45.156745+0000","last_unstale":"2024-03-30T13:08:49.641749+0000","last_undegraded":"2024-03-30T13:08:49.641749+0000","last_fullsized":"2024-03-30T13:08:49.641749+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:08:41.532076+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:08:41.532076+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:43.071429+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":57731253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":155,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-03-30T13:08:48.211082+0000","last_change":"2024-03-30T13:07:53.686045+0000","last_active":"2024-03-30T13:08:48.211082+0000","last_peered":"2024-03-30T13:08:48.211082+0000","last_clean":"2024-03-30T13:08:48.211082+0000","last_became_active":"2024-03-30T13:07:45.154855+0000","last_became_peered":"2024-03-30T13:07:45.154855+0000","last_unstale":"2024-03-30T13:08:48.211082+0000","last_undegraded":"2024-03-30T13:08:48.211082+0000","last_fullsized":"2024-03-30T13:08:48.211082+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:07:53.685778+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:58.028417+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":317,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-03-30T13:08:47.757102+0000","last_change":"2024-03-30T13:08:24.700616+0000","last_active":"2024-03-30T13:08:47.757102+0000","last_peered":"2024-03-30T13:08:47.757102+0000","last_clean":"2024-03-30T13:08:47.757102+0000","last_became_active":"2024-03-30T13:06:54.153682+0000","last_became_peered":"2024-03-30T13:06:54.153682+0000","last_unstale":"2024-03-30T13:08:47.757102+0000","last_undegraded":"2024-03-30T13:08:47.757102+0000","last_fullsized":"2024-03-30T13:08:47.757102+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:08:24.700361+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:08:24.700361+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:54.523068+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":163,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-03-30T13:08:48.211120+0000","last_change":"2024-03-30T13:08:07.461164+0000","last_active":"2024-03-30T13:08:48.211120+0000","last_peered":"2024-03-30T13:08:48.211120+0000","last_clean":"2024-03-30T13:08:48.211120+0000","last_became_active":"2024-03-30T13:07:45.154381+0000","last_became_peered":"2024-03-30T13:07:45.154381+0000","last_unstale":"2024-03-30T13:08:48.211120+0000","last_undegraded":"2024-03-30T13:08:48.211120+0000","last_fullsized":"2024-03-30T13:08:48.211120+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:08:07.460876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:08:07.460876+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:20.973897+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":75,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-03-30T13:08:48.640675+0000","last_change":"2024-03-30T13:08:39.544104+0000","last_active":"2024-03-30T13:08:48.640675+0000","last_peered":"2024-03-30T13:08:48.640675+0000","last_clean":"2024-03-30T13:08:48.640675+0000","last_became_active":"2024-03-30T13:08:39.543465+0000","last_became_peered":"2024-03-30T13:08:39.543465+0000","last_unstale":"2024-03-30T13:08:48.640675+0000","last_undegraded":"2024-03-30T13:08:48.640675+0000","last_fullsized":"2024-03-30T13:08:48.640675+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:38.221640+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:07:38.221640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":300521875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":227,"seq":974957576250,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51060,"kb_used_data":19368,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94320780,"statfs":{"total":96636764160,"available":96584478720,"internally_reserved":0,"allocated":19832832,"data_stored":17427818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534735,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56708,"kb_used_data":24112,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94315132,"statfs":{"total":96636764160,"available":96578695168,"internally_reserved":0,"allocated":24690688,"data_stored":22285674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":33350120},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824047,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50384,"kb_used_data":17792,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94321456,"statfs":{"total":96636764160,"available":96585170944,"internally_reserved":0,"allocated":18219008,"data_stored":15800458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509603,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41780,"kb_used_data":10596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330060,"statfs":{"total":96636764160,"available":96593981440,"internally_reserved":0,"allocated":10850304,"data_stored":8438522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":346,"seq":1486058684421,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40060,"kb_used_data":7920,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94331780,"statfs":{"total":96636764160,"available":96595742720,"internally_reserved":0,"allocated":8110080,"data_stored":5698298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6045,"internal_metadata":32893027},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509603,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48940,"kb_used_data":17760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322900,"statfs":{"total":96636764160,"available":96586649600,"internally_reserved":0,"allocated":18186240,"data_stored":15781226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":298,"seq":1279900254233,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47776,"kb_used_data":14292,"kb_used_omap":10,"kb_used_meta":33461,"kb_avail":94324064,"statfs":{"total":96636764160,"available":96587841536,"internally_reserved":0,"allocated":14635008,"data_stored":12236762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10986,"internal_metadata":34264342},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575011,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47928,"kb_used_data":16748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323912,"statfs":{"total":96636764160,"available":96587685888,"internally_reserved":0,"allocated":17149952,"data_stored":14751706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3735552,"data_stored":3735552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":13832192,"data_stored":13832192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:08:52.356 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:08:52.356 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:08:52.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:52.469+0000 7f51c3da9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:52.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:52.481+0000 7f51c3da9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:52.655 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:08:52.655 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":363,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:08:51.662029+0000","last_up_change":"2024-03-30T13:08:34.478709+0000","last_in_change":"2024-03-30T13:08:35.493374+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":83,"last_clean_end":336,"up_from":346,"up_thru":350,"down_at":338,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2287832000},{"type":"v1","addr":"172.21.15.67:6818","nonce":2287832000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2287832000},{"type":"v1","addr":"172.21.15.67:6821","nonce":2287832000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2287832000},{"type":"v1","addr":"172.21.15.67:6828","nonce":2287832000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2287832000},{"type":"v1","addr":"172.21.15.67:6825","nonce":2287832000}]},"public_addr":"172.21.15.67:6818/2287832000","cluster_addr":"172.21.15.67:6821/2287832000","heartbeat_back_addr":"172.21.15.67:6828/2287832000","heartbeat_front_addr":"172.21.15.67:6825/2287832000","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":349,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":320,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6801","nonce":1504348724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6803","nonce":1504348724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6807","nonce":1504348724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6805","nonce":1504348724}]},"public_addr":"172.21.15.67:6801/1504348724","cluster_addr":"172.21.15.67:6803/1504348724","heartbeat_back_addr":"172.21.15.67:6807/1504348724","heartbeat_front_addr":"172.21.15.67:6805/1504348724","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":135,"last_clean_end":243,"up_from":250,"up_thru":348,"down_at":244,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6803","nonce":3986522908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6807","nonce":3986522908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6815","nonce":3986522908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6811","nonce":3986522908}]},"public_addr":"172.21.15.82:6803/3986522908","cluster_addr":"172.21.15.82:6807/3986522908","heartbeat_back_addr":"172.21.15.82:6815/3986522908","heartbeat_front_addr":"172.21.15.82:6811/3986522908","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":274,"last_clean_end":314,"up_from":320,"up_thru":320,"down_at":315,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":925466951},{"type":"v1","addr":"172.21.15.82:6817","nonce":925466951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":925466951},{"type":"v1","addr":"172.21.15.82:6819","nonce":925466951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":925466951},{"type":"v1","addr":"172.21.15.82:6823","nonce":925466951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":925466951},{"type":"v1","addr":"172.21.15.82:6821","nonce":925466951}]},"public_addr":"172.21.15.82:6817/925466951","cluster_addr":"172.21.15.82:6819/925466951","heartbeat_back_addr":"172.21.15.82:6823/925466951","heartbeat_front_addr":"172.21.15.82:6821/925466951","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":15,"last_clean_end":222,"up_from":227,"up_thru":320,"down_at":223,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3068584906},{"type":"v1","addr":"172.21.15.82:6801","nonce":3068584906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3068584906},{"type":"v1","addr":"172.21.15.82:6806","nonce":3068584906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3068584906},{"type":"v1","addr":"172.21.15.82:6814","nonce":3068584906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3068584906},{"type":"v1","addr":"172.21.15.82:6810","nonce":3068584906}]},"public_addr":"172.21.15.82:6801/3068584906","cluster_addr":"172.21.15.82:6806/3068584906","heartbeat_back_addr":"172.21.15.82:6814/3068584906","heartbeat_front_addr":"172.21.15.82:6810/3068584906","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":206,"last_clean_end":292,"up_from":298,"up_thru":350,"down_at":293,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3894469331},{"type":"v1","addr":"172.21.15.82:6825","nonce":3894469331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3894469331},{"type":"v1","addr":"172.21.15.82:6827","nonce":3894469331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3894469331},{"type":"v1","addr":"172.21.15.82:6831","nonce":3894469331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3894469331},{"type":"v1","addr":"172.21.15.82:6829","nonce":3894469331}]},"public_addr":"172.21.15.82:6825/3894469331","cluster_addr":"172.21.15.82:6827/3894469331","heartbeat_back_addr":"172.21.15.82:6831/3894469331","heartbeat_front_addr":"172.21.15.82:6829/3894469331","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:08:09.546736+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:08:34.393279+0000","dead_epoch":338},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:07:57.620702+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:08:29.898816+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:08:15.840398+0000","dead_epoch":244},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:08:14.267114+0000","dead_epoch":315},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:08:01.338058+0000","dead_epoch":223},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:08:07.519450+0000","dead_epoch":293}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"1.3","osds":[4,6,5]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:08:52.676 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:08:52.676 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:08:52.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:52.793+0000 7fb5da984640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:52.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:52.809+0000 7fb5da984640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:52.979 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:08:52.998 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:08:53.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:53.117+0000 7ff3f85fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:53.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:53.133+0000 7ff3f85fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:53.314 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:08:53.315 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:08:53.333 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:08:53.333 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:08:53.333 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:08:53.333 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:08:53.334 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 3, 6, 4, 7, 5, 1] 2024-03-30T13:08:53.334 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:08:53.334 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:08:53.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:53.453+0000 7f6c7d52f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:53.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:53.465+0000 7f6c7d52f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:53.631 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:08:53.632 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:08:53.648 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":756,"stamp":"2024-03-30T13:08:52.011175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":300521875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":49,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":384636,"kb_used_data":128588,"kb_used_omap":40,"kb_used_meta":255895,"kb_avail":754590084,"statfs":{"total":773094113280,"available":772700246016,"internally_reserved":0,"allocated":131674112,"data_stored":112420464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41615,"internal_metadata":262036849},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001164"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":816,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-30T13:08:49.641841+0000","last_change":"2024-03-30T13:07:45.158080+0000","last_active":"2024-03-30T13:08:49.641841+0000","last_peered":"2024-03-30T13:08:49.641841+0000","last_clean":"2024-03-30T13:08:49.641841+0000","last_became_active":"2024-03-30T13:07:45.156785+0000","last_became_peered":"2024-03-30T13:07:45.156785+0000","last_unstale":"2024-03-30T13:08:49.641841+0000","last_undegraded":"2024-03-30T13:08:49.641841+0000","last_fullsized":"2024-03-30T13:08:49.641841+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:38.717555+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:07:38.717555+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:57.207615+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22941225,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":767,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-30T13:08:50.123856+0000","last_change":"2024-03-30T13:08:32.414902+0000","last_active":"2024-03-30T13:08:50.123856+0000","last_peered":"2024-03-30T13:08:50.123856+0000","last_clean":"2024-03-30T13:08:50.123856+0000","last_became_active":"2024-03-30T13:06:54.153354+0000","last_became_peered":"2024-03-30T13:06:54.153354+0000","last_unstale":"2024-03-30T13:08:50.123856+0000","last_undegraded":"2024-03-30T13:08:50.123856+0000","last_fullsized":"2024-03-30T13:08:50.123856+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:08:32.414658+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:08:32.414658+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:38.583836+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":400,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-03-30T13:08:47.757223+0000","last_change":"2024-03-30T13:08:39.542911+0000","last_active":"2024-03-30T13:08:47.757223+0000","last_peered":"2024-03-30T13:08:47.757223+0000","last_clean":"2024-03-30T13:08:47.757223+0000","last_became_active":"2024-03-30T13:08:39.541896+0000","last_became_peered":"2024-03-30T13:08:39.541896+0000","last_unstale":"2024-03-30T13:08:47.757223+0000","last_undegraded":"2024-03-30T13:08:47.757223+0000","last_fullsized":"2024-03-30T13:08:47.757223+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:08:37.747009+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:43.320096+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":352,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-03-30T13:08:47.761786+0000","last_change":"2024-03-30T13:08:47.761868+0000","last_active":"2024-03-30T13:08:47.761786+0000","last_peered":"2024-03-30T13:08:47.761786+0000","last_clean":"2024-03-30T13:08:47.761786+0000","last_became_active":"2024-03-30T13:06:54.156927+0000","last_became_peered":"2024-03-30T13:06:54.156927+0000","last_unstale":"2024-03-30T13:08:47.761786+0000","last_undegraded":"2024-03-30T13:08:47.761786+0000","last_fullsized":"2024-03-30T13:08:47.761786+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:08:47.761560+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:08:47.761560+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:53.763939+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":139,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-03-30T13:08:48.211091+0000","last_change":"2024-03-30T13:07:45.158822+0000","last_active":"2024-03-30T13:08:48.211091+0000","last_peered":"2024-03-30T13:08:48.211091+0000","last_clean":"2024-03-30T13:08:48.211091+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:08:48.211091+0000","last_undegraded":"2024-03-30T13:08:48.211091+0000","last_fullsized":"2024-03-30T13:08:48.211091+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:07:37.576416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:07:37.576416+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:50.550338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":851,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-03-30T13:08:47.624949+0000","last_change":"2024-03-30T13:07:45.157963+0000","last_active":"2024-03-30T13:08:47.624949+0000","last_peered":"2024-03-30T13:08:47.624949+0000","last_clean":"2024-03-30T13:08:47.624949+0000","last_became_active":"2024-03-30T13:07:45.156174+0000","last_became_peered":"2024-03-30T13:07:45.156174+0000","last_unstale":"2024-03-30T13:08:47.624949+0000","last_undegraded":"2024-03-30T13:08:47.624949+0000","last_fullsized":"2024-03-30T13:08:47.624949+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_clean_scrub_stamp":"2024-03-30T13:07:40.613260+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:54.625356+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":56974905,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":379,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-03-30T13:08:47.757115+0000","last_change":"2024-03-30T13:08:26.697182+0000","last_active":"2024-03-30T13:08:47.757115+0000","last_peered":"2024-03-30T13:08:47.757115+0000","last_clean":"2024-03-30T13:08:47.757115+0000","last_became_active":"2024-03-30T13:06:54.155351+0000","last_became_peered":"2024-03-30T13:06:54.155351+0000","last_unstale":"2024-03-30T13:08:47.757115+0000","last_undegraded":"2024-03-30T13:08:47.757115+0000","last_fullsized":"2024-03-30T13:08:47.757115+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:08:26.696919+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:08:26.696919+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:38.403041+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":69,"reported_epoch":361,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-30T13:08:50.123472+0000","last_change":"2024-03-30T13:08:37.516659+0000","last_active":"2024-03-30T13:08:50.123472+0000","last_peered":"2024-03-30T13:08:50.123472+0000","last_clean":"2024-03-30T13:08:34.571093+0000","last_became_active":"2024-03-30T13:08:37.514196+0000","last_became_peered":"2024-03-30T13:08:37.514196+0000","last_unstale":"2024-03-30T13:08:50.123472+0000","last_undegraded":"2024-03-30T13:08:50.123472+0000","last_fullsized":"2024-03-30T13:08:50.123472+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:07:44.108131+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:07:44.108131+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[4,6,5],"avail_no_missing":["4(0)","1(0)","5(2)","6(1)"],"object_location_counts":[{"shards":"4(0),5(2),6(1)","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":930,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-03-30T13:08:47.625172+0000","last_change":"2024-03-30T13:08:29.496007+0000","last_active":"2024-03-30T13:08:47.625172+0000","last_peered":"2024-03-30T13:08:47.625172+0000","last_clean":"2024-03-30T13:08:47.625172+0000","last_became_active":"2024-03-30T13:07:45.156128+0000","last_became_peered":"2024-03-30T13:07:45.156128+0000","last_unstale":"2024-03-30T13:08:47.625172+0000","last_undegraded":"2024-03-30T13:08:47.625172+0000","last_fullsized":"2024-03-30T13:08:47.625172+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:08:29.495729+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:08:29.495729+0000","last_clean_scrub_stamp":"2024-03-30T13:08:29.495729+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:32.078835+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":65983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":669,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-30T13:08:50.123472+0000","last_change":"2024-03-30T13:08:04.573877+0000","last_active":"2024-03-30T13:08:50.123472+0000","last_peered":"2024-03-30T13:08:50.123472+0000","last_clean":"2024-03-30T13:08:50.123472+0000","last_became_active":"2024-03-30T13:05:07.445793+0000","last_became_peered":"2024-03-30T13:05:07.445793+0000","last_unstale":"2024-03-30T13:08:50.123472+0000","last_undegraded":"2024-03-30T13:08:50.123472+0000","last_fullsized":"2024-03-30T13:08:50.123472+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:08:04.573631+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:08:04.573631+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:27.130867+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":829,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-30T13:08:49.641674+0000","last_change":"2024-03-30T13:08:49.546393+0000","last_active":"2024-03-30T13:08:49.641674+0000","last_peered":"2024-03-30T13:08:49.641674+0000","last_clean":"2024-03-30T13:08:49.641674+0000","last_became_active":"2024-03-30T13:05:07.445475+0000","last_became_peered":"2024-03-30T13:05:07.445475+0000","last_unstale":"2024-03-30T13:08:49.641674+0000","last_undegraded":"2024-03-30T13:08:49.641674+0000","last_fullsized":"2024-03-30T13:08:49.641674+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:08:49.546103+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:08:49.546103+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:50.816946+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":878,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-30T13:08:49.641749+0000","last_change":"2024-03-30T13:08:41.532360+0000","last_active":"2024-03-30T13:08:49.641749+0000","last_peered":"2024-03-30T13:08:49.641749+0000","last_clean":"2024-03-30T13:08:49.641749+0000","last_became_active":"2024-03-30T13:07:45.156745+0000","last_became_peered":"2024-03-30T13:07:45.156745+0000","last_unstale":"2024-03-30T13:08:49.641749+0000","last_undegraded":"2024-03-30T13:08:49.641749+0000","last_fullsized":"2024-03-30T13:08:49.641749+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:08:41.532076+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:08:41.532076+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:43.071429+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":57731253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":155,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-03-30T13:08:48.211082+0000","last_change":"2024-03-30T13:07:53.686045+0000","last_active":"2024-03-30T13:08:48.211082+0000","last_peered":"2024-03-30T13:08:48.211082+0000","last_clean":"2024-03-30T13:08:48.211082+0000","last_became_active":"2024-03-30T13:07:45.154855+0000","last_became_peered":"2024-03-30T13:07:45.154855+0000","last_unstale":"2024-03-30T13:08:48.211082+0000","last_undegraded":"2024-03-30T13:08:48.211082+0000","last_fullsized":"2024-03-30T13:08:48.211082+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:07:53.685778+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:58.028417+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":317,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-03-30T13:08:47.757102+0000","last_change":"2024-03-30T13:08:24.700616+0000","last_active":"2024-03-30T13:08:47.757102+0000","last_peered":"2024-03-30T13:08:47.757102+0000","last_clean":"2024-03-30T13:08:47.757102+0000","last_became_active":"2024-03-30T13:06:54.153682+0000","last_became_peered":"2024-03-30T13:06:54.153682+0000","last_unstale":"2024-03-30T13:08:47.757102+0000","last_undegraded":"2024-03-30T13:08:47.757102+0000","last_fullsized":"2024-03-30T13:08:47.757102+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:08:24.700361+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:08:24.700361+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:54.523068+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":163,"reported_epoch":359,"state":"active+clean","last_fresh":"2024-03-30T13:08:48.211120+0000","last_change":"2024-03-30T13:08:07.461164+0000","last_active":"2024-03-30T13:08:48.211120+0000","last_peered":"2024-03-30T13:08:48.211120+0000","last_clean":"2024-03-30T13:08:48.211120+0000","last_became_active":"2024-03-30T13:07:45.154381+0000","last_became_peered":"2024-03-30T13:07:45.154381+0000","last_unstale":"2024-03-30T13:08:48.211120+0000","last_undegraded":"2024-03-30T13:08:48.211120+0000","last_fullsized":"2024-03-30T13:08:48.211120+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:08:07.460876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:08:07.460876+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:20.973897+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":75,"reported_epoch":360,"state":"active+clean","last_fresh":"2024-03-30T13:08:48.640675+0000","last_change":"2024-03-30T13:08:39.544104+0000","last_active":"2024-03-30T13:08:48.640675+0000","last_peered":"2024-03-30T13:08:48.640675+0000","last_clean":"2024-03-30T13:08:48.640675+0000","last_became_active":"2024-03-30T13:08:39.543465+0000","last_became_peered":"2024-03-30T13:08:39.543465+0000","last_unstale":"2024-03-30T13:08:48.640675+0000","last_undegraded":"2024-03-30T13:08:48.640675+0000","last_fullsized":"2024-03-30T13:08:48.640675+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:38.221640+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:07:38.221640+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":300521875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":227,"seq":974957576250,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51060,"kb_used_data":19368,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94320780,"statfs":{"total":96636764160,"available":96584478720,"internally_reserved":0,"allocated":19832832,"data_stored":17427818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534735,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56708,"kb_used_data":24112,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94315132,"statfs":{"total":96636764160,"available":96578695168,"internally_reserved":0,"allocated":24690688,"data_stored":22285674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":33350120},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824047,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50384,"kb_used_data":17792,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94321456,"statfs":{"total":96636764160,"available":96585170944,"internally_reserved":0,"allocated":18219008,"data_stored":15800458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509603,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41780,"kb_used_data":10596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330060,"statfs":{"total":96636764160,"available":96593981440,"internally_reserved":0,"allocated":10850304,"data_stored":8438522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":346,"seq":1486058684421,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40060,"kb_used_data":7920,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94331780,"statfs":{"total":96636764160,"available":96595742720,"internally_reserved":0,"allocated":8110080,"data_stored":5698298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6045,"internal_metadata":32893027},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509603,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48940,"kb_used_data":17760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322900,"statfs":{"total":96636764160,"available":96586649600,"internally_reserved":0,"allocated":18186240,"data_stored":15781226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":298,"seq":1279900254233,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47776,"kb_used_data":14292,"kb_used_omap":10,"kb_used_meta":33461,"kb_avail":94324064,"statfs":{"total":96636764160,"available":96587841536,"internally_reserved":0,"allocated":14635008,"data_stored":12236762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10986,"internal_metadata":34264342},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575011,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47928,"kb_used_data":16748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323912,"statfs":{"total":96636764160,"available":96587685888,"internally_reserved":0,"allocated":17149952,"data_stored":14751706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3735552,"data_stored":3735552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":13832192,"data_stored":13832192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:08:53.649 INFO:tasks.thrashosds.thrasher:Choosing PG 1.6 with acting set [6, 0, 4] 2024-03-30T13:08:53.649 INFO:tasks.thrashosds.thrasher:Killing osd 6, live_osds are [0, 2, 3, 6, 4, 7, 5, 1] 2024-03-30T13:08:53.649 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-30T13:08:53.649 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:08:53.707 INFO:tasks.ceph.osd.6:Stopped 2024-03-30T13:08:53.708 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 2, 3, 6, 4, 7, 5, 1] 2024-03-30T13:08:53.708 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 6 2024-03-30T13:08:53.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:53.813+0000 7f6505c55640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:53.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:53.829+0000 7f6505c55640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:54.008 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:08:54.009+0000 7fbb38f08640 -1 mon.a@0(leader).osd e363 definitely_dead 0 2024-03-30T13:08:54.704 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.6. 2024-03-30T13:08:54.729 INFO:tasks.thrashosds.thrasher:dead_osds=[6], live_osds=[0, 2, 3, 4, 7, 5, 1] 2024-03-30T13:08:54.729 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:08:54.730 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:08:54.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:54.849+0000 7f2d77b74640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:54.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:08:54.861+0000 7f2d77b74640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:08:55.037 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:08:55.037 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:08:55.055 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":758,"stamp":"2024-03-30T13:08:54.693665+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":300521875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":333780,"kb_used_data":109424,"kb_used_omap":36,"kb_used_meta":224219,"kb_avail":660269100,"statfs":{"total":676457349120,"available":676115558400,"internally_reserved":0,"allocated":112050176,"data_stored":95086486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37217,"internal_metadata":229600927},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.054045"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":816,"reported_epoch":361,"state":"stale+active+clean","last_fresh":"2024-03-30T13:08:49.641841+0000","last_change":"2024-03-30T13:07:45.158080+0000","last_active":"2024-03-30T13:08:49.641841+0000","last_peered":"2024-03-30T13:08:49.641841+0000","last_clean":"2024-03-30T13:08:49.641841+0000","last_became_active":"2024-03-30T13:07:45.156785+0000","last_became_peered":"2024-03-30T13:07:45.156785+0000","last_unstale":"2024-03-30T13:08:54.693695+0000","last_undegraded":"2024-03-30T13:08:49.641841+0000","last_fullsized":"2024-03-30T13:08:49.641841+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:07:38.717555+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:07:38.717555+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:57.207615+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22941225,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":767,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-30T13:08:50.123856+0000","last_change":"2024-03-30T13:08:32.414902+0000","last_active":"2024-03-30T13:08:50.123856+0000","last_peered":"2024-03-30T13:08:50.123856+0000","last_clean":"2024-03-30T13:08:50.123856+0000","last_became_active":"2024-03-30T13:06:54.153354+0000","last_became_peered":"2024-03-30T13:06:54.153354+0000","last_unstale":"2024-03-30T13:08:50.123856+0000","last_undegraded":"2024-03-30T13:08:50.123856+0000","last_fullsized":"2024-03-30T13:08:50.123856+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:08:32.414658+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:08:32.414658+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:38.583836+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":407,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-30T13:08:51.802717+0000","last_change":"2024-03-30T13:08:39.542911+0000","last_active":"2024-03-30T13:08:51.802717+0000","last_peered":"2024-03-30T13:08:51.802717+0000","last_clean":"2024-03-30T13:08:51.802717+0000","last_became_active":"2024-03-30T13:08:39.541896+0000","last_became_peered":"2024-03-30T13:08:39.541896+0000","last_unstale":"2024-03-30T13:08:51.802717+0000","last_undegraded":"2024-03-30T13:08:51.802717+0000","last_fullsized":"2024-03-30T13:08:51.802717+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:08:37.747009+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:43.320096+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":360,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-30T13:08:51.802742+0000","last_change":"2024-03-30T13:08:47.761868+0000","last_active":"2024-03-30T13:08:51.802742+0000","last_peered":"2024-03-30T13:08:51.802742+0000","last_clean":"2024-03-30T13:08:51.802742+0000","last_became_active":"2024-03-30T13:06:54.156927+0000","last_became_peered":"2024-03-30T13:06:54.156927+0000","last_unstale":"2024-03-30T13:08:51.802742+0000","last_undegraded":"2024-03-30T13:08:51.802742+0000","last_fullsized":"2024-03-30T13:08:51.802742+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:08:47.761560+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:08:47.761560+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:53.763939+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":154,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-30T13:08:53.435317+0000","last_change":"2024-03-30T13:08:53.435394+0000","last_active":"2024-03-30T13:08:53.435317+0000","last_peered":"2024-03-30T13:08:53.435317+0000","last_clean":"2024-03-30T13:08:53.435317+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:08:53.435317+0000","last_undegraded":"2024-03-30T13:08:53.435317+0000","last_fullsized":"2024-03-30T13:08:53.435317+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:08:53.435125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:08:53.435125+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:55.081351+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":857,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-30T13:08:51.677112+0000","last_change":"2024-03-30T13:07:45.157963+0000","last_active":"2024-03-30T13:08:51.677112+0000","last_peered":"2024-03-30T13:08:51.677112+0000","last_clean":"2024-03-30T13:08:51.677112+0000","last_became_active":"2024-03-30T13:07:45.156174+0000","last_became_peered":"2024-03-30T13:07:45.156174+0000","last_unstale":"2024-03-30T13:08:51.677112+0000","last_undegraded":"2024-03-30T13:08:51.677112+0000","last_fullsized":"2024-03-30T13:08:51.677112+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_clean_scrub_stamp":"2024-03-30T13:07:40.613260+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:54.625356+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":56974905,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":386,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-30T13:08:51.802756+0000","last_change":"2024-03-30T13:08:26.697182+0000","last_active":"2024-03-30T13:08:51.802756+0000","last_peered":"2024-03-30T13:08:51.802756+0000","last_clean":"2024-03-30T13:08:51.802756+0000","last_became_active":"2024-03-30T13:06:54.155351+0000","last_became_peered":"2024-03-30T13:06:54.155351+0000","last_unstale":"2024-03-30T13:08:51.802756+0000","last_undegraded":"2024-03-30T13:08:51.802756+0000","last_fullsized":"2024-03-30T13:08:51.802756+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:08:26.696919+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:08:26.696919+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:38.403041+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":69,"reported_epoch":361,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-30T13:08:50.123472+0000","last_change":"2024-03-30T13:08:37.516659+0000","last_active":"2024-03-30T13:08:50.123472+0000","last_peered":"2024-03-30T13:08:50.123472+0000","last_clean":"2024-03-30T13:08:34.571093+0000","last_became_active":"2024-03-30T13:08:37.514196+0000","last_became_peered":"2024-03-30T13:08:37.514196+0000","last_unstale":"2024-03-30T13:08:50.123472+0000","last_undegraded":"2024-03-30T13:08:50.123472+0000","last_fullsized":"2024-03-30T13:08:50.123472+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:07:44.108131+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:07:44.108131+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[4,6,5],"avail_no_missing":["4(0)","1(0)","5(2)","6(1)"],"object_location_counts":[{"shards":"4(0),5(2),6(1)","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":936,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-30T13:08:51.677183+0000","last_change":"2024-03-30T13:08:29.496007+0000","last_active":"2024-03-30T13:08:51.677183+0000","last_peered":"2024-03-30T13:08:51.677183+0000","last_clean":"2024-03-30T13:08:51.677183+0000","last_became_active":"2024-03-30T13:07:45.156128+0000","last_became_peered":"2024-03-30T13:07:45.156128+0000","last_unstale":"2024-03-30T13:08:51.677183+0000","last_undegraded":"2024-03-30T13:08:51.677183+0000","last_fullsized":"2024-03-30T13:08:51.677183+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:08:29.495729+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:08:29.495729+0000","last_clean_scrub_stamp":"2024-03-30T13:08:29.495729+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:32.078835+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":65983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":669,"reported_epoch":361,"state":"active+clean","last_fresh":"2024-03-30T13:08:50.123472+0000","last_change":"2024-03-30T13:08:04.573877+0000","last_active":"2024-03-30T13:08:50.123472+0000","last_peered":"2024-03-30T13:08:50.123472+0000","last_clean":"2024-03-30T13:08:50.123472+0000","last_became_active":"2024-03-30T13:05:07.445793+0000","last_became_peered":"2024-03-30T13:05:07.445793+0000","last_unstale":"2024-03-30T13:08:50.123472+0000","last_undegraded":"2024-03-30T13:08:50.123472+0000","last_fullsized":"2024-03-30T13:08:50.123472+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:08:04.573631+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:08:04.573631+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:27.130867+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":829,"reported_epoch":361,"state":"stale+active+clean","last_fresh":"2024-03-30T13:08:49.641674+0000","last_change":"2024-03-30T13:08:49.546393+0000","last_active":"2024-03-30T13:08:49.641674+0000","last_peered":"2024-03-30T13:08:49.641674+0000","last_clean":"2024-03-30T13:08:49.641674+0000","last_became_active":"2024-03-30T13:05:07.445475+0000","last_became_peered":"2024-03-30T13:05:07.445475+0000","last_unstale":"2024-03-30T13:08:54.693701+0000","last_undegraded":"2024-03-30T13:08:49.641674+0000","last_fullsized":"2024-03-30T13:08:49.641674+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:08:49.546103+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:08:49.546103+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:50.816946+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":878,"reported_epoch":361,"state":"stale+active+clean","last_fresh":"2024-03-30T13:08:49.641749+0000","last_change":"2024-03-30T13:08:41.532360+0000","last_active":"2024-03-30T13:08:49.641749+0000","last_peered":"2024-03-30T13:08:49.641749+0000","last_clean":"2024-03-30T13:08:49.641749+0000","last_became_active":"2024-03-30T13:07:45.156745+0000","last_became_peered":"2024-03-30T13:07:45.156745+0000","last_unstale":"2024-03-30T13:08:54.693706+0000","last_undegraded":"2024-03-30T13:08:49.641749+0000","last_fullsized":"2024-03-30T13:08:49.641749+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:08:41.532076+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:08:41.532076+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:43.071429+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":57731253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":162,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-30T13:08:51.801862+0000","last_change":"2024-03-30T13:07:53.686045+0000","last_active":"2024-03-30T13:08:51.801862+0000","last_peered":"2024-03-30T13:08:51.801862+0000","last_clean":"2024-03-30T13:08:51.801862+0000","last_became_active":"2024-03-30T13:07:45.154855+0000","last_became_peered":"2024-03-30T13:07:45.154855+0000","last_unstale":"2024-03-30T13:08:51.801862+0000","last_undegraded":"2024-03-30T13:08:51.801862+0000","last_fullsized":"2024-03-30T13:08:51.801862+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:07:53.685778+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:08:58.028417+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":324,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-30T13:08:51.802598+0000","last_change":"2024-03-30T13:08:24.700616+0000","last_active":"2024-03-30T13:08:51.802598+0000","last_peered":"2024-03-30T13:08:51.802598+0000","last_clean":"2024-03-30T13:08:51.802598+0000","last_became_active":"2024-03-30T13:06:54.153682+0000","last_became_peered":"2024-03-30T13:06:54.153682+0000","last_unstale":"2024-03-30T13:08:51.802598+0000","last_undegraded":"2024-03-30T13:08:51.802598+0000","last_fullsized":"2024-03-30T13:08:51.802598+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:08:24.700361+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:08:24.700361+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:54.523068+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":170,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-30T13:08:51.801890+0000","last_change":"2024-03-30T13:08:07.461164+0000","last_active":"2024-03-30T13:08:51.801890+0000","last_peered":"2024-03-30T13:08:51.801890+0000","last_clean":"2024-03-30T13:08:51.801890+0000","last_became_active":"2024-03-30T13:07:45.154381+0000","last_became_peered":"2024-03-30T13:07:45.154381+0000","last_unstale":"2024-03-30T13:08:51.801890+0000","last_undegraded":"2024-03-30T13:08:51.801890+0000","last_fullsized":"2024-03-30T13:08:51.801890+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:08:07.460876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:08:07.460876+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:20.973897+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":89,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-30T13:08:52.434196+0000","last_change":"2024-03-30T13:08:52.434273+0000","last_active":"2024-03-30T13:08:52.434196+0000","last_peered":"2024-03-30T13:08:52.434196+0000","last_clean":"2024-03-30T13:08:52.434196+0000","last_became_active":"2024-03-30T13:08:39.543465+0000","last_became_peered":"2024-03-30T13:08:39.543465+0000","last_unstale":"2024-03-30T13:08:52.434196+0000","last_undegraded":"2024-03-30T13:08:52.434196+0000","last_fullsized":"2024-03-30T13:08:52.434196+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:08:52.434054+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:08:52.434054+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:05.803298+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124,"num_bytes_recovered":300521875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534736,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56748,"kb_used_data":24160,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94315092,"statfs":{"total":96636764160,"available":96578654208,"internally_reserved":0,"allocated":24739840,"data_stored":22307754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":33350120},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824047,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50384,"kb_used_data":17792,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94321456,"statfs":{"total":96636764160,"available":96585170944,"internally_reserved":0,"allocated":18219008,"data_stored":15800458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509603,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41780,"kb_used_data":10596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330060,"statfs":{"total":96636764160,"available":96593981440,"internally_reserved":0,"allocated":10850304,"data_stored":8438522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":346,"seq":1486058684422,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40096,"kb_used_data":7956,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94331744,"statfs":{"total":96636764160,"available":96595705856,"internally_reserved":0,"allocated":8146944,"data_stored":5714858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6045,"internal_metadata":32893027},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509603,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48940,"kb_used_data":17760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322900,"statfs":{"total":96636764160,"available":96586649600,"internally_reserved":0,"allocated":18186240,"data_stored":15781226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":298,"seq":1279900254234,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47844,"kb_used_data":14352,"kb_used_omap":10,"kb_used_meta":33461,"kb_avail":94323996,"statfs":{"total":96636764160,"available":96587771904,"internally_reserved":0,"allocated":14696448,"data_stored":12264362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10986,"internal_metadata":34264342},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575012,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47988,"kb_used_data":16808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323852,"statfs":{"total":96636764160,"available":96587624448,"internally_reserved":0,"allocated":17211392,"data_stored":14779306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3735552,"data_stored":3735552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":13832192,"data_stored":13832192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:08:55.056 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:09:20.057 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:09:20.058 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:09:20.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:20.188+0000 7f3fd1990640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:20.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:20.200+0000 7f3fd1990640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:20.377 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:09:20.378 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:09:20.395 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":777,"stamp":"2024-03-30T13:09:20.019016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":259681426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":354940,"kb_used_data":126432,"kb_used_omap":36,"kb_used_meta":228379,"kb_avail":660247940,"statfs":{"total":676457349120,"available":676093890560,"internally_reserved":0,"allocated":129466368,"data_stored":112118622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37217,"internal_metadata":233860767},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002245"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":34,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-03-30T13:09:16.180272+0000","last_change":"2024-03-30T13:08:59.482199+0000","last_active":"2024-03-30T13:09:16.180272+0000","last_peered":"2024-03-30T13:09:16.180272+0000","last_clean":"2024-03-30T13:09:16.180272+0000","last_became_active":"2024-03-30T13:08:55.711751+0000","last_became_peered":"2024-03-30T13:08:55.711751+0000","last_unstale":"2024-03-30T13:09:16.180272+0000","last_undegraded":"2024-03-30T13:09:16.180272+0000","last_fullsized":"2024-03-30T13:09:16.180272+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:08:59.481960+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:08:59.481960+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:18.376769+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":784,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-03-30T13:09:16.116987+0000","last_change":"2024-03-30T13:08:32.414902+0000","last_active":"2024-03-30T13:09:16.116987+0000","last_peered":"2024-03-30T13:09:16.116987+0000","last_clean":"2024-03-30T13:09:16.116987+0000","last_became_active":"2024-03-30T13:06:54.153354+0000","last_became_peered":"2024-03-30T13:06:54.153354+0000","last_unstale":"2024-03-30T13:09:16.116987+0000","last_undegraded":"2024-03-30T13:09:16.116987+0000","last_fullsized":"2024-03-30T13:09:16.116987+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:08:32.414658+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:08:32.414658+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:38.583836+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":421,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-03-30T13:09:16.177451+0000","last_change":"2024-03-30T13:08:39.542911+0000","last_active":"2024-03-30T13:09:16.177451+0000","last_peered":"2024-03-30T13:09:16.177451+0000","last_clean":"2024-03-30T13:09:16.177451+0000","last_became_active":"2024-03-30T13:08:39.541896+0000","last_became_peered":"2024-03-30T13:08:39.541896+0000","last_unstale":"2024-03-30T13:09:16.177451+0000","last_undegraded":"2024-03-30T13:09:16.177451+0000","last_fullsized":"2024-03-30T13:09:16.177451+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:08:37.747009+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:43.320096+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":385,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-03-30T13:09:16.176790+0000","last_change":"2024-03-30T13:08:55.836012+0000","last_active":"2024-03-30T13:09:16.176790+0000","last_peered":"2024-03-30T13:09:16.176790+0000","last_clean":"2024-03-30T13:09:16.176790+0000","last_became_active":"2024-03-30T13:08:55.713340+0000","last_became_peered":"2024-03-30T13:08:55.713340+0000","last_unstale":"2024-03-30T13:09:16.176790+0000","last_undegraded":"2024-03-30T13:09:16.176790+0000","last_fullsized":"2024-03-30T13:09:16.176790+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:08:47.761560+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:08:47.761560+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:55.948122+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":168,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-03-30T13:09:16.176737+0000","last_change":"2024-03-30T13:08:53.435394+0000","last_active":"2024-03-30T13:09:16.176737+0000","last_peered":"2024-03-30T13:09:16.176737+0000","last_clean":"2024-03-30T13:09:16.176737+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:09:16.176737+0000","last_undegraded":"2024-03-30T13:09:16.176737+0000","last_fullsized":"2024-03-30T13:09:16.176737+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:08:53.435125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:08:53.435125+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:55.081351+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":879,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-03-30T13:09:16.180370+0000","last_change":"2024-03-30T13:09:01.452554+0000","last_active":"2024-03-30T13:09:16.180370+0000","last_peered":"2024-03-30T13:09:16.180370+0000","last_clean":"2024-03-30T13:09:16.180370+0000","last_became_active":"2024-03-30T13:07:45.156174+0000","last_became_peered":"2024-03-30T13:07:45.156174+0000","last_unstale":"2024-03-30T13:09:16.180370+0000","last_undegraded":"2024-03-30T13:09:16.180370+0000","last_fullsized":"2024-03-30T13:09:16.180370+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:09:01.452292+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_clean_scrub_stamp":"2024-03-30T13:09:01.452292+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:03.443879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":56974905,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":400,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-03-30T13:09:16.177160+0000","last_change":"2024-03-30T13:08:26.697182+0000","last_active":"2024-03-30T13:09:16.177160+0000","last_peered":"2024-03-30T13:09:16.177160+0000","last_clean":"2024-03-30T13:09:16.177160+0000","last_became_active":"2024-03-30T13:06:54.155351+0000","last_became_peered":"2024-03-30T13:06:54.155351+0000","last_unstale":"2024-03-30T13:09:16.177160+0000","last_undegraded":"2024-03-30T13:09:16.177160+0000","last_fullsized":"2024-03-30T13:09:16.177160+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:08:26.696919+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:08:26.696919+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:38.403041+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":122,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-03-30T13:09:16.118932+0000","last_change":"2024-03-30T13:08:59.518206+0000","last_active":"2024-03-30T13:09:16.118932+0000","last_peered":"2024-03-30T13:09:16.118932+0000","last_clean":"2024-03-30T13:09:16.118932+0000","last_became_active":"2024-03-30T13:08:58.734359+0000","last_became_peered":"2024-03-30T13:08:58.734359+0000","last_unstale":"2024-03-30T13:09:16.118932+0000","last_undegraded":"2024-03-30T13:09:16.118932+0000","last_fullsized":"2024-03-30T13:09:16.118932+0000","mapping_epoch":367,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":368,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:08:59.517955+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:08:59.517955+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:06.161550+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":949,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-03-30T13:09:16.180377+0000","last_change":"2024-03-30T13:08:29.496007+0000","last_active":"2024-03-30T13:09:16.180377+0000","last_peered":"2024-03-30T13:09:16.180377+0000","last_clean":"2024-03-30T13:09:16.180377+0000","last_became_active":"2024-03-30T13:07:45.156128+0000","last_became_peered":"2024-03-30T13:07:45.156128+0000","last_unstale":"2024-03-30T13:09:16.180377+0000","last_undegraded":"2024-03-30T13:09:16.180377+0000","last_fullsized":"2024-03-30T13:09:16.180377+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:08:29.495729+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:08:29.495729+0000","last_clean_scrub_stamp":"2024-03-30T13:08:29.495729+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:32.078835+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":65983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":705,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-03-30T13:09:16.117144+0000","last_change":"2024-03-30T13:09:15.464660+0000","last_active":"2024-03-30T13:09:16.117144+0000","last_peered":"2024-03-30T13:09:16.117144+0000","last_clean":"2024-03-30T13:09:16.117144+0000","last_became_active":"2024-03-30T13:08:55.709885+0000","last_became_peered":"2024-03-30T13:08:55.709885+0000","last_unstale":"2024-03-30T13:09:16.117144+0000","last_undegraded":"2024-03-30T13:09:16.117144+0000","last_fullsized":"2024-03-30T13:09:16.117144+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:15.464410+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:09:15.464410+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:35.583682+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":32,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-03-30T13:09:16.177666+0000","last_change":"2024-03-30T13:08:55.839557+0000","last_active":"2024-03-30T13:09:16.177666+0000","last_peered":"2024-03-30T13:09:16.177666+0000","last_clean":"2024-03-30T13:09:16.177666+0000","last_became_active":"2024-03-30T13:08:55.710867+0000","last_became_peered":"2024-03-30T13:08:55.710867+0000","last_unstale":"2024-03-30T13:09:16.177666+0000","last_undegraded":"2024-03-30T13:09:16.177666+0000","last_fullsized":"2024-03-30T13:09:16.177666+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:08:49.546103+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:08:49.546103+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:14.481192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":38,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-03-30T13:09:16.177713+0000","last_change":"2024-03-30T13:08:55.982877+0000","last_active":"2024-03-30T13:09:16.177713+0000","last_peered":"2024-03-30T13:09:16.177713+0000","last_clean":"2024-03-30T13:09:16.177713+0000","last_became_active":"2024-03-30T13:08:55.714607+0000","last_became_peered":"2024-03-30T13:08:55.714607+0000","last_unstale":"2024-03-30T13:09:16.177713+0000","last_undegraded":"2024-03-30T13:09:16.177713+0000","last_fullsized":"2024-03-30T13:09:16.177713+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:08:41.532076+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:08:41.532076+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:56.562521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":184,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-03-30T13:09:16.176727+0000","last_change":"2024-03-30T13:09:00.517090+0000","last_active":"2024-03-30T13:09:16.176727+0000","last_peered":"2024-03-30T13:09:16.176727+0000","last_clean":"2024-03-30T13:09:16.176727+0000","last_became_active":"2024-03-30T13:07:45.154855+0000","last_became_peered":"2024-03-30T13:07:45.154855+0000","last_unstale":"2024-03-30T13:09:16.176727+0000","last_undegraded":"2024-03-30T13:09:16.176727+0000","last_fullsized":"2024-03-30T13:09:16.176727+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:09:00.516843+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:09:00.516843+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:15.153201+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":338,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-03-30T13:09:16.176955+0000","last_change":"2024-03-30T13:08:24.700616+0000","last_active":"2024-03-30T13:09:16.176955+0000","last_peered":"2024-03-30T13:09:16.176955+0000","last_clean":"2024-03-30T13:09:16.176955+0000","last_became_active":"2024-03-30T13:06:54.153682+0000","last_became_peered":"2024-03-30T13:06:54.153682+0000","last_unstale":"2024-03-30T13:09:16.176955+0000","last_undegraded":"2024-03-30T13:09:16.176955+0000","last_fullsized":"2024-03-30T13:09:16.176955+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:08:24.700361+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:08:24.700361+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:54.523068+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":200,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-03-30T13:09:16.176773+0000","last_change":"2024-03-30T13:08:55.794009+0000","last_active":"2024-03-30T13:09:16.176773+0000","last_peered":"2024-03-30T13:09:16.176773+0000","last_clean":"2024-03-30T13:09:16.176773+0000","last_became_active":"2024-03-30T13:08:55.711290+0000","last_became_peered":"2024-03-30T13:08:55.711290+0000","last_unstale":"2024-03-30T13:09:16.176773+0000","last_undegraded":"2024-03-30T13:09:16.176773+0000","last_fullsized":"2024-03-30T13:09:16.176773+0000","mapping_epoch":364,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":365,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:08:07.460876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:08:07.460876+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:32.007156+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":103,"reported_epoch":370,"state":"active+clean","last_fresh":"2024-03-30T13:09:16.119149+0000","last_change":"2024-03-30T13:08:52.434273+0000","last_active":"2024-03-30T13:09:16.119149+0000","last_peered":"2024-03-30T13:09:16.119149+0000","last_clean":"2024-03-30T13:09:16.119149+0000","last_became_active":"2024-03-30T13:08:39.543465+0000","last_became_peered":"2024-03-30T13:08:39.543465+0000","last_unstale":"2024-03-30T13:09:16.119149+0000","last_undegraded":"2024-03-30T13:09:16.119149+0000","last_fullsized":"2024-03-30T13:09:16.119149+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:08:52.434054+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:08:52.434054+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:05.803298+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":259681426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":113467392,"data_stored":113467392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534741,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56832,"kb_used_data":24244,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94315008,"statfs":{"total":96636764160,"available":96578568192,"internally_reserved":0,"allocated":24825856,"data_stored":22346641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":33350120},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824052,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48524,"kb_used_data":15936,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94323316,"statfs":{"total":96636764160,"available":96587075584,"internally_reserved":0,"allocated":16318464,"data_stored":13846008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509609,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43216,"kb_used_data":12032,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328624,"statfs":{"total":96636764160,"available":96592510976,"internally_reserved":0,"allocated":12320768,"data_stored":9841553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":346,"seq":1486058684427,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42132,"kb_used_data":9992,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94329708,"statfs":{"total":96636764160,"available":96593620992,"internally_reserved":0,"allocated":10231808,"data_stored":7752593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6045,"internal_metadata":32893027},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509609,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54172,"kb_used_data":22992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94317668,"statfs":{"total":96636764160,"available":96581292032,"internally_reserved":0,"allocated":23543808,"data_stored":21064593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":298,"seq":1279900254239,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60960,"kb_used_data":23316,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94310880,"statfs":{"total":96636764160,"available":96574341120,"internally_reserved":0,"allocated":23875584,"data_stored":21396369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10986,"internal_metadata":38524182},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575017,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49104,"kb_used_data":17920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322736,"statfs":{"total":96636764160,"available":96586481664,"internally_reserved":0,"allocated":18350080,"data_stored":15870865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19046400,"data_stored":19046400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7823360,"data_stored":7823360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":13852672,"data_stored":13852672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":19378176,"data_stored":19378176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:09:20.396 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:09:20.396 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:09:20.396 INFO:tasks.thrashosds.thrasher:Reviving osd 6 2024-03-30T13:09:20.396 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-30T13:09:20.396 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-30T13:09:20.399 INFO:tasks.ceph.osd.6:Started 2024-03-30T13:09:20.399 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-30T13:09:20.465 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T13:09:20.466+0000 7f8a06e307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:20.472 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T13:09:20.474+0000 7f8a06e307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:20.473 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T13:09:20.474+0000 7f8a06e307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:20.519 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T13:09:20.519 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T13:09:20.520 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T13:09:20.520 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T13:09:20.520 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T13:09:20.520 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T13:09:20.520 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-03-30T13:09:20.520 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-03-30T13:09:20.520 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-03-30T13:09:20.520 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-03-30T13:09:20.520 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-03-30T13:09:20.520 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T13:09:20.522 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:09:20.522 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-03-30T13:09:21.010 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T13:09:21.010+0000 7f8a06e307c0 -1 Falling back to public interface 2024-03-30T13:09:24.810 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T13:09:24.810+0000 7f8a06e307c0 -1 osd.6 363 log_to_monitors true 2024-03-30T13:09:25.177 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T13:09:25.178+0000 7f89fec05640 -1 osd.6 363 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:09:25.523 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-30T13:09:25.656 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T13:09:25.656 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T13:09:25.657 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T13:09:25.657 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T13:09:25.670 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:09:25.670 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-03-30T13:09:25.670 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 6 2024-03-30T13:09:25.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:25.800+0000 7f5b23026640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:25.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:25.816+0000 7f5b23026640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:25.996 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:09:25.996+0000 7fbb38f08640 -1 mon.a@0(leader).osd e371 definitely_dead 0 2024-03-30T13:09:26.152 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.6. 2024-03-30T13:09:26.179 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-03-30T13:09:26.180 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:09:26.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:26.296+0000 7f071a9cd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:26.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:26.312+0000 7f071a9cd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:27.143 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:09:27.176 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:09:27.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:27.296+0000 7f99a212a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:27.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:27.312+0000 7f99a212a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:28.168 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:09:28.193 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:09:28.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:28.312+0000 7fc2a5996640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:28.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:28.328+0000 7fc2a5996640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:29.176 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:09:29.200 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:09:29.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:29.316+0000 7ff85f944640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:29.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:29.332+0000 7ff85f944640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:30.185 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:09:30.210 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:09:30.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:30.332+0000 7f04186f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:30.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:30.348+0000 7f04186f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:31.188 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:09:31.219 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:09:31.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:31.340+0000 7f5de5717640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:31.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:31.356+0000 7f5de5717640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:32.211 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:09:32.233 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:09:32.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:32.352+0000 7f1a40eb7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:32.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:32.368+0000 7f1a40eb7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:33.207 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:09:33.232 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:09:33.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:33.352+0000 7f3a432f2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:33.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:33.368+0000 7f3a432f2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:09:34.253 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:09:34.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:34.376+0000 7fdcc4940640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:34.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:34.392+0000 7fdcc4940640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:35.237 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:09:35.259 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:09:35.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:35.380+0000 7ff9aba76640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:35.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:35.396+0000 7ff9aba76640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:36.246 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:09:36.274 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:09:36.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:36.396+0000 7efd2dc1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:36.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:36.412+0000 7efd2dc1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:37.257 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:09:37.282 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:09:37.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:37.400+0000 7fe3362e5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:37.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:37.416+0000 7fe3362e5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:38.270 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:09:38.298 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:09:38.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:38.420+0000 7f8a47b1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:38.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:38.436+0000 7f8a47b1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:39.275 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:09:39.304 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:09:39.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:39.424+0000 7fb7c9463640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:39.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:39.436+0000 7fb7c9463640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:40.296 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:09:40.320 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:09:40.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:40.439+0000 7f3dea207640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:40.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:40.455+0000 7f3dea207640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:41.312 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:09:41.340 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:09:41.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:41.459+0000 7f0defbce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:41.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:41.475+0000 7f0defbce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:42.321 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:09:42.347 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:09:42.348 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:09:42.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:42.467+0000 7fa21a95d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:42.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:42.479+0000 7fa21a95d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:42.655 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:09:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:09:42.673 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":806,"stamp":"2024-03-30T13:09:42.318928+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":227698134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":391612,"kb_used_data":130940,"kb_used_omap":42,"kb_used_meta":260501,"kb_avail":754583108,"statfs":{"total":773094113280,"available":772693102592,"internally_reserved":0,"allocated":134082560,"data_stored":113515098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43263,"internal_metadata":266753793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.044329"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":857,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-30T13:09:40.530809+0000","last_change":"2024-03-30T13:09:27.171258+0000","last_active":"2024-03-30T13:09:40.530809+0000","last_peered":"2024-03-30T13:09:40.530809+0000","last_clean":"2024-03-30T13:09:40.530809+0000","last_became_active":"2024-03-30T13:09:27.168768+0000","last_became_peered":"2024-03-30T13:09:27.168768+0000","last_unstale":"2024-03-30T13:09:40.530809+0000","last_undegraded":"2024-03-30T13:09:40.530809+0000","last_fullsized":"2024-03-30T13:09:40.530809+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:08:59.481960+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:08:59.481960+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:13.508213+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":826,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-30T13:09:41.318922+0000","last_change":"2024-03-30T13:09:39.429119+0000","last_active":"2024-03-30T13:09:41.318922+0000","last_peered":"2024-03-30T13:09:41.318922+0000","last_clean":"2024-03-30T13:09:41.318922+0000","last_became_active":"2024-03-30T13:06:54.153354+0000","last_became_peered":"2024-03-30T13:06:54.153354+0000","last_unstale":"2024-03-30T13:09:41.318922+0000","last_undegraded":"2024-03-30T13:09:41.318922+0000","last_fullsized":"2024-03-30T13:09:41.318922+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:39.428863+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:09:39.428863+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:08.015563+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":445,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-30T13:09:36.635789+0000","last_change":"2024-03-30T13:08:39.542911+0000","last_active":"2024-03-30T13:09:36.635789+0000","last_peered":"2024-03-30T13:09:36.635789+0000","last_clean":"2024-03-30T13:09:36.635789+0000","last_became_active":"2024-03-30T13:08:39.541896+0000","last_became_peered":"2024-03-30T13:08:39.541896+0000","last_unstale":"2024-03-30T13:09:36.635789+0000","last_undegraded":"2024-03-30T13:09:36.635789+0000","last_fullsized":"2024-03-30T13:09:36.635789+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:08:37.747009+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:43.320096+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":415,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-30T13:09:36.635703+0000","last_change":"2024-03-30T13:09:27.165474+0000","last_active":"2024-03-30T13:09:36.635703+0000","last_peered":"2024-03-30T13:09:36.635703+0000","last_clean":"2024-03-30T13:09:36.635703+0000","last_became_active":"2024-03-30T13:09:27.164268+0000","last_became_peered":"2024-03-30T13:09:27.164268+0000","last_unstale":"2024-03-30T13:09:36.635703+0000","last_undegraded":"2024-03-30T13:09:36.635703+0000","last_fullsized":"2024-03-30T13:09:36.635703+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:08:47.761560+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:08:47.761560+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:48.582641+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":193,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-30T13:09:37.629370+0000","last_change":"2024-03-30T13:08:53.435394+0000","last_active":"2024-03-30T13:09:37.629370+0000","last_peered":"2024-03-30T13:09:37.629370+0000","last_clean":"2024-03-30T13:09:37.629370+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:09:37.629370+0000","last_undegraded":"2024-03-30T13:09:37.629370+0000","last_fullsized":"2024-03-30T13:09:37.629370+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:08:53.435125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:08:53.435125+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:55.081351+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":903,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-30T13:09:36.608271+0000","last_change":"2024-03-30T13:09:01.452554+0000","last_active":"2024-03-30T13:09:36.608271+0000","last_peered":"2024-03-30T13:09:36.608271+0000","last_clean":"2024-03-30T13:09:36.608271+0000","last_became_active":"2024-03-30T13:07:45.156174+0000","last_became_peered":"2024-03-30T13:07:45.156174+0000","last_unstale":"2024-03-30T13:09:36.608271+0000","last_undegraded":"2024-03-30T13:09:36.608271+0000","last_fullsized":"2024-03-30T13:09:36.608271+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:09:01.452292+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_clean_scrub_stamp":"2024-03-30T13:09:01.452292+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:03.443879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":56974905,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":424,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-30T13:09:36.635702+0000","last_change":"2024-03-30T13:08:26.697182+0000","last_active":"2024-03-30T13:09:36.635702+0000","last_peered":"2024-03-30T13:09:36.635702+0000","last_clean":"2024-03-30T13:09:36.635702+0000","last_became_active":"2024-03-30T13:06:54.155351+0000","last_became_peered":"2024-03-30T13:06:54.155351+0000","last_unstale":"2024-03-30T13:09:36.635702+0000","last_undegraded":"2024-03-30T13:09:36.635702+0000","last_fullsized":"2024-03-30T13:09:36.635702+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:08:26.696919+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:08:26.696919+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:38.403041+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":158,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-30T13:09:39.284954+0000","last_change":"2024-03-30T13:09:27.547106+0000","last_active":"2024-03-30T13:09:39.284954+0000","last_peered":"2024-03-30T13:09:39.284954+0000","last_clean":"2024-03-30T13:09:39.284954+0000","last_became_active":"2024-03-30T13:09:27.546829+0000","last_became_peered":"2024-03-30T13:09:27.546829+0000","last_unstale":"2024-03-30T13:09:39.284954+0000","last_undegraded":"2024-03-30T13:09:39.284954+0000","last_fullsized":"2024-03-30T13:09:39.284954+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:08:59.517955+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:08:59.517955+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:21.998387+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":981,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-30T13:09:36.608295+0000","last_change":"2024-03-30T13:09:34.162066+0000","last_active":"2024-03-30T13:09:36.608295+0000","last_peered":"2024-03-30T13:09:36.608295+0000","last_clean":"2024-03-30T13:09:36.608295+0000","last_became_active":"2024-03-30T13:07:45.156128+0000","last_became_peered":"2024-03-30T13:07:45.156128+0000","last_unstale":"2024-03-30T13:09:36.608295+0000","last_undegraded":"2024-03-30T13:09:36.608295+0000","last_fullsized":"2024-03-30T13:09:36.608295+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:09:34.161806+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:08:29.495729+0000","last_clean_scrub_stamp":"2024-03-30T13:09:34.161806+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:39.085627+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":65983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":744,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-30T13:09:41.318709+0000","last_change":"2024-03-30T13:09:27.170807+0000","last_active":"2024-03-30T13:09:41.318709+0000","last_peered":"2024-03-30T13:09:41.318709+0000","last_clean":"2024-03-30T13:09:41.318709+0000","last_became_active":"2024-03-30T13:09:27.169549+0000","last_became_peered":"2024-03-30T13:09:27.169549+0000","last_unstale":"2024-03-30T13:09:41.318709+0000","last_undegraded":"2024-03-30T13:09:41.318709+0000","last_fullsized":"2024-03-30T13:09:41.318709+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:15.464410+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:09:15.464410+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:39.896795+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":864,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-30T13:09:40.530339+0000","last_change":"2024-03-30T13:09:27.168331+0000","last_active":"2024-03-30T13:09:40.530339+0000","last_peered":"2024-03-30T13:09:40.530339+0000","last_clean":"2024-03-30T13:09:40.530339+0000","last_became_active":"2024-03-30T13:09:27.167070+0000","last_became_peered":"2024-03-30T13:09:27.167070+0000","last_unstale":"2024-03-30T13:09:40.530339+0000","last_undegraded":"2024-03-30T13:09:40.530339+0000","last_fullsized":"2024-03-30T13:09:40.530339+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:08:49.546103+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:08:49.546103+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:53.421805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":899,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-30T13:09:40.530596+0000","last_change":"2024-03-30T13:09:27.170410+0000","last_active":"2024-03-30T13:09:40.530596+0000","last_peered":"2024-03-30T13:09:40.530596+0000","last_clean":"2024-03-30T13:09:40.530596+0000","last_became_active":"2024-03-30T13:09:27.168910+0000","last_became_peered":"2024-03-30T13:09:27.168910+0000","last_unstale":"2024-03-30T13:09:40.530596+0000","last_undegraded":"2024-03-30T13:09:40.530596+0000","last_fullsized":"2024-03-30T13:09:40.530596+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:08:41.532076+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:08:41.532076+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:01.657802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":209,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-30T13:09:37.629403+0000","last_change":"2024-03-30T13:09:00.517090+0000","last_active":"2024-03-30T13:09:37.629403+0000","last_peered":"2024-03-30T13:09:37.629403+0000","last_clean":"2024-03-30T13:09:37.629403+0000","last_became_active":"2024-03-30T13:07:45.154855+0000","last_became_peered":"2024-03-30T13:07:45.154855+0000","last_unstale":"2024-03-30T13:09:37.629403+0000","last_undegraded":"2024-03-30T13:09:37.629403+0000","last_fullsized":"2024-03-30T13:09:37.629403+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:09:00.516843+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:09:00.516843+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:15.153201+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":362,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-30T13:09:36.635700+0000","last_change":"2024-03-30T13:08:24.700616+0000","last_active":"2024-03-30T13:09:36.635700+0000","last_peered":"2024-03-30T13:09:36.635700+0000","last_clean":"2024-03-30T13:09:36.635700+0000","last_became_active":"2024-03-30T13:06:54.153682+0000","last_became_peered":"2024-03-30T13:06:54.153682+0000","last_unstale":"2024-03-30T13:09:36.635700+0000","last_undegraded":"2024-03-30T13:09:36.635700+0000","last_fullsized":"2024-03-30T13:09:36.635700+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:08:24.700361+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:08:24.700361+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:54.523068+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":239,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-30T13:09:37.629624+0000","last_change":"2024-03-30T13:09:34.558439+0000","last_active":"2024-03-30T13:09:37.629624+0000","last_peered":"2024-03-30T13:09:37.629624+0000","last_clean":"2024-03-30T13:09:37.629624+0000","last_became_active":"2024-03-30T13:09:27.162655+0000","last_became_peered":"2024-03-30T13:09:27.162655+0000","last_unstale":"2024-03-30T13:09:37.629624+0000","last_undegraded":"2024-03-30T13:09:37.629624+0000","last_fullsized":"2024-03-30T13:09:37.629624+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:09:34.558164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:09:34.558164+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:45.791448+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":133,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-30T13:09:39.285082+0000","last_change":"2024-03-30T13:08:52.434273+0000","last_active":"2024-03-30T13:09:39.285082+0000","last_peered":"2024-03-30T13:09:39.285082+0000","last_clean":"2024-03-30T13:09:39.285082+0000","last_became_active":"2024-03-30T13:08:39.543465+0000","last_became_peered":"2024-03-30T13:08:39.543465+0000","last_unstale":"2024-03-30T13:09:39.285082+0000","last_undegraded":"2024-03-30T13:09:39.285082+0000","last_fullsized":"2024-03-30T13:09:39.285082+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:08:52.434054+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:08:52.434054+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:05.803298+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":227698134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834117,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51832,"kb_used_data":19692,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94320008,"statfs":{"total":96636764160,"available":96583688192,"internally_reserved":0,"allocated":20164608,"data_stored":17578373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534745,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56988,"kb_used_data":24400,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94314852,"statfs":{"total":96636764160,"available":96578408448,"internally_reserved":0,"allocated":24985600,"data_stored":22419768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":33350120},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824057,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48732,"kb_used_data":16140,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94323108,"statfs":{"total":96636764160,"available":96586862592,"internally_reserved":0,"allocated":16527360,"data_stored":13941125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509613,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42056,"kb_used_data":10872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329784,"statfs":{"total":96636764160,"available":96593698816,"internally_reserved":0,"allocated":11132928,"data_stored":8567096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":346,"seq":1486058684431,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42304,"kb_used_data":10160,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94329536,"statfs":{"total":96636764160,"available":96593444864,"internally_reserved":0,"allocated":10403840,"data_stored":7831207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6045,"internal_metadata":32893027},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509613,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49244,"kb_used_data":18060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322596,"statfs":{"total":96636764160,"available":96586338304,"internally_reserved":0,"allocated":18493440,"data_stored":15920807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":298,"seq":1279900254243,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52236,"kb_used_data":14580,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94319604,"statfs":{"total":96636764160,"available":96583274496,"internally_reserved":0,"allocated":14929920,"data_stored":12370889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10986,"internal_metadata":38524182},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575021,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48220,"kb_used_data":17036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323620,"statfs":{"total":96636764160,"available":96587386880,"internally_reserved":0,"allocated":17444864,"data_stored":14885833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:09:42.675 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:09:42.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:42.791+0000 7f9d91ab4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:42.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:42.807+0000 7f9d91ab4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:42.978 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:09:42.979 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:09:42.998 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":806,"stamp":"2024-03-30T13:09:42.318928+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":227698134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":391612,"kb_used_data":130940,"kb_used_omap":42,"kb_used_meta":260501,"kb_avail":754583108,"statfs":{"total":773094113280,"available":772693102592,"internally_reserved":0,"allocated":134082560,"data_stored":113515098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43263,"internal_metadata":266753793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.044329"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":857,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-30T13:09:40.530809+0000","last_change":"2024-03-30T13:09:27.171258+0000","last_active":"2024-03-30T13:09:40.530809+0000","last_peered":"2024-03-30T13:09:40.530809+0000","last_clean":"2024-03-30T13:09:40.530809+0000","last_became_active":"2024-03-30T13:09:27.168768+0000","last_became_peered":"2024-03-30T13:09:27.168768+0000","last_unstale":"2024-03-30T13:09:40.530809+0000","last_undegraded":"2024-03-30T13:09:40.530809+0000","last_fullsized":"2024-03-30T13:09:40.530809+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:08:59.481960+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:08:59.481960+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:13.508213+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":826,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-30T13:09:41.318922+0000","last_change":"2024-03-30T13:09:39.429119+0000","last_active":"2024-03-30T13:09:41.318922+0000","last_peered":"2024-03-30T13:09:41.318922+0000","last_clean":"2024-03-30T13:09:41.318922+0000","last_became_active":"2024-03-30T13:06:54.153354+0000","last_became_peered":"2024-03-30T13:06:54.153354+0000","last_unstale":"2024-03-30T13:09:41.318922+0000","last_undegraded":"2024-03-30T13:09:41.318922+0000","last_fullsized":"2024-03-30T13:09:41.318922+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:39.428863+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:09:39.428863+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:08.015563+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":445,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-30T13:09:36.635789+0000","last_change":"2024-03-30T13:08:39.542911+0000","last_active":"2024-03-30T13:09:36.635789+0000","last_peered":"2024-03-30T13:09:36.635789+0000","last_clean":"2024-03-30T13:09:36.635789+0000","last_became_active":"2024-03-30T13:08:39.541896+0000","last_became_peered":"2024-03-30T13:08:39.541896+0000","last_unstale":"2024-03-30T13:09:36.635789+0000","last_undegraded":"2024-03-30T13:09:36.635789+0000","last_fullsized":"2024-03-30T13:09:36.635789+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:08:37.747009+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:43.320096+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":415,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-30T13:09:36.635703+0000","last_change":"2024-03-30T13:09:27.165474+0000","last_active":"2024-03-30T13:09:36.635703+0000","last_peered":"2024-03-30T13:09:36.635703+0000","last_clean":"2024-03-30T13:09:36.635703+0000","last_became_active":"2024-03-30T13:09:27.164268+0000","last_became_peered":"2024-03-30T13:09:27.164268+0000","last_unstale":"2024-03-30T13:09:36.635703+0000","last_undegraded":"2024-03-30T13:09:36.635703+0000","last_fullsized":"2024-03-30T13:09:36.635703+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:08:47.761560+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:08:47.761560+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:48.582641+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":193,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-30T13:09:37.629370+0000","last_change":"2024-03-30T13:08:53.435394+0000","last_active":"2024-03-30T13:09:37.629370+0000","last_peered":"2024-03-30T13:09:37.629370+0000","last_clean":"2024-03-30T13:09:37.629370+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:09:37.629370+0000","last_undegraded":"2024-03-30T13:09:37.629370+0000","last_fullsized":"2024-03-30T13:09:37.629370+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:08:53.435125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:08:53.435125+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:55.081351+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":903,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-30T13:09:36.608271+0000","last_change":"2024-03-30T13:09:01.452554+0000","last_active":"2024-03-30T13:09:36.608271+0000","last_peered":"2024-03-30T13:09:36.608271+0000","last_clean":"2024-03-30T13:09:36.608271+0000","last_became_active":"2024-03-30T13:07:45.156174+0000","last_became_peered":"2024-03-30T13:07:45.156174+0000","last_unstale":"2024-03-30T13:09:36.608271+0000","last_undegraded":"2024-03-30T13:09:36.608271+0000","last_fullsized":"2024-03-30T13:09:36.608271+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:09:01.452292+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_clean_scrub_stamp":"2024-03-30T13:09:01.452292+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:03.443879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":56974905,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":424,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-30T13:09:36.635702+0000","last_change":"2024-03-30T13:08:26.697182+0000","last_active":"2024-03-30T13:09:36.635702+0000","last_peered":"2024-03-30T13:09:36.635702+0000","last_clean":"2024-03-30T13:09:36.635702+0000","last_became_active":"2024-03-30T13:06:54.155351+0000","last_became_peered":"2024-03-30T13:06:54.155351+0000","last_unstale":"2024-03-30T13:09:36.635702+0000","last_undegraded":"2024-03-30T13:09:36.635702+0000","last_fullsized":"2024-03-30T13:09:36.635702+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:08:26.696919+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:08:26.696919+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:38.403041+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":158,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-30T13:09:39.284954+0000","last_change":"2024-03-30T13:09:27.547106+0000","last_active":"2024-03-30T13:09:39.284954+0000","last_peered":"2024-03-30T13:09:39.284954+0000","last_clean":"2024-03-30T13:09:39.284954+0000","last_became_active":"2024-03-30T13:09:27.546829+0000","last_became_peered":"2024-03-30T13:09:27.546829+0000","last_unstale":"2024-03-30T13:09:39.284954+0000","last_undegraded":"2024-03-30T13:09:39.284954+0000","last_fullsized":"2024-03-30T13:09:39.284954+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:08:59.517955+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:08:59.517955+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:21.998387+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":981,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-30T13:09:36.608295+0000","last_change":"2024-03-30T13:09:34.162066+0000","last_active":"2024-03-30T13:09:36.608295+0000","last_peered":"2024-03-30T13:09:36.608295+0000","last_clean":"2024-03-30T13:09:36.608295+0000","last_became_active":"2024-03-30T13:07:45.156128+0000","last_became_peered":"2024-03-30T13:07:45.156128+0000","last_unstale":"2024-03-30T13:09:36.608295+0000","last_undegraded":"2024-03-30T13:09:36.608295+0000","last_fullsized":"2024-03-30T13:09:36.608295+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:09:34.161806+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:08:29.495729+0000","last_clean_scrub_stamp":"2024-03-30T13:09:34.161806+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:39.085627+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":65983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":744,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-30T13:09:41.318709+0000","last_change":"2024-03-30T13:09:27.170807+0000","last_active":"2024-03-30T13:09:41.318709+0000","last_peered":"2024-03-30T13:09:41.318709+0000","last_clean":"2024-03-30T13:09:41.318709+0000","last_became_active":"2024-03-30T13:09:27.169549+0000","last_became_peered":"2024-03-30T13:09:27.169549+0000","last_unstale":"2024-03-30T13:09:41.318709+0000","last_undegraded":"2024-03-30T13:09:41.318709+0000","last_fullsized":"2024-03-30T13:09:41.318709+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:15.464410+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:09:15.464410+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:39.896795+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":864,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-30T13:09:40.530339+0000","last_change":"2024-03-30T13:09:27.168331+0000","last_active":"2024-03-30T13:09:40.530339+0000","last_peered":"2024-03-30T13:09:40.530339+0000","last_clean":"2024-03-30T13:09:40.530339+0000","last_became_active":"2024-03-30T13:09:27.167070+0000","last_became_peered":"2024-03-30T13:09:27.167070+0000","last_unstale":"2024-03-30T13:09:40.530339+0000","last_undegraded":"2024-03-30T13:09:40.530339+0000","last_fullsized":"2024-03-30T13:09:40.530339+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:08:49.546103+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:08:49.546103+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:53.421805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":899,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-30T13:09:40.530596+0000","last_change":"2024-03-30T13:09:27.170410+0000","last_active":"2024-03-30T13:09:40.530596+0000","last_peered":"2024-03-30T13:09:40.530596+0000","last_clean":"2024-03-30T13:09:40.530596+0000","last_became_active":"2024-03-30T13:09:27.168910+0000","last_became_peered":"2024-03-30T13:09:27.168910+0000","last_unstale":"2024-03-30T13:09:40.530596+0000","last_undegraded":"2024-03-30T13:09:40.530596+0000","last_fullsized":"2024-03-30T13:09:40.530596+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:08:41.532076+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:08:41.532076+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:01.657802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":209,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-30T13:09:37.629403+0000","last_change":"2024-03-30T13:09:00.517090+0000","last_active":"2024-03-30T13:09:37.629403+0000","last_peered":"2024-03-30T13:09:37.629403+0000","last_clean":"2024-03-30T13:09:37.629403+0000","last_became_active":"2024-03-30T13:07:45.154855+0000","last_became_peered":"2024-03-30T13:07:45.154855+0000","last_unstale":"2024-03-30T13:09:37.629403+0000","last_undegraded":"2024-03-30T13:09:37.629403+0000","last_fullsized":"2024-03-30T13:09:37.629403+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:09:00.516843+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:09:00.516843+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:15.153201+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":362,"reported_epoch":382,"state":"active+clean","last_fresh":"2024-03-30T13:09:36.635700+0000","last_change":"2024-03-30T13:08:24.700616+0000","last_active":"2024-03-30T13:09:36.635700+0000","last_peered":"2024-03-30T13:09:36.635700+0000","last_clean":"2024-03-30T13:09:36.635700+0000","last_became_active":"2024-03-30T13:06:54.153682+0000","last_became_peered":"2024-03-30T13:06:54.153682+0000","last_unstale":"2024-03-30T13:09:36.635700+0000","last_undegraded":"2024-03-30T13:09:36.635700+0000","last_fullsized":"2024-03-30T13:09:36.635700+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:08:24.700361+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:08:24.700361+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:54.523068+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":239,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-30T13:09:37.629624+0000","last_change":"2024-03-30T13:09:34.558439+0000","last_active":"2024-03-30T13:09:37.629624+0000","last_peered":"2024-03-30T13:09:37.629624+0000","last_clean":"2024-03-30T13:09:37.629624+0000","last_became_active":"2024-03-30T13:09:27.162655+0000","last_became_peered":"2024-03-30T13:09:27.162655+0000","last_unstale":"2024-03-30T13:09:37.629624+0000","last_undegraded":"2024-03-30T13:09:37.629624+0000","last_fullsized":"2024-03-30T13:09:37.629624+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:09:34.558164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:09:34.558164+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:45.791448+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":133,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-30T13:09:39.285082+0000","last_change":"2024-03-30T13:08:52.434273+0000","last_active":"2024-03-30T13:09:39.285082+0000","last_peered":"2024-03-30T13:09:39.285082+0000","last_clean":"2024-03-30T13:09:39.285082+0000","last_became_active":"2024-03-30T13:08:39.543465+0000","last_became_peered":"2024-03-30T13:08:39.543465+0000","last_unstale":"2024-03-30T13:09:39.285082+0000","last_undegraded":"2024-03-30T13:09:39.285082+0000","last_fullsized":"2024-03-30T13:09:39.285082+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:08:52.434054+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:08:52.434054+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:05.803298+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":227698134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834117,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51832,"kb_used_data":19692,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94320008,"statfs":{"total":96636764160,"available":96583688192,"internally_reserved":0,"allocated":20164608,"data_stored":17578373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534745,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56988,"kb_used_data":24400,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94314852,"statfs":{"total":96636764160,"available":96578408448,"internally_reserved":0,"allocated":24985600,"data_stored":22419768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":33350120},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824057,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48732,"kb_used_data":16140,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94323108,"statfs":{"total":96636764160,"available":96586862592,"internally_reserved":0,"allocated":16527360,"data_stored":13941125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509613,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42056,"kb_used_data":10872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329784,"statfs":{"total":96636764160,"available":96593698816,"internally_reserved":0,"allocated":11132928,"data_stored":8567096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":346,"seq":1486058684431,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42304,"kb_used_data":10160,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94329536,"statfs":{"total":96636764160,"available":96593444864,"internally_reserved":0,"allocated":10403840,"data_stored":7831207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6045,"internal_metadata":32893027},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509613,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49244,"kb_used_data":18060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322596,"statfs":{"total":96636764160,"available":96586338304,"internally_reserved":0,"allocated":18493440,"data_stored":15920807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":298,"seq":1279900254243,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52236,"kb_used_data":14580,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94319604,"statfs":{"total":96636764160,"available":96583274496,"internally_reserved":0,"allocated":14929920,"data_stored":12370889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10986,"internal_metadata":38524182},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575021,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48220,"kb_used_data":17036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323620,"statfs":{"total":96636764160,"available":96587386880,"internally_reserved":0,"allocated":17444864,"data_stored":14885833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:09:42.999 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:09:42.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:09:43.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:43.111+0000 7fe278dfc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:43.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:43.127+0000 7fe278dfc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:43.294 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:09:43.294 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":388,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:09:42.316851+0000","last_up_change":"2024-03-30T13:09:26.142515+0000","last_in_change":"2024-03-30T13:09:26.142515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":21,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":83,"last_clean_end":336,"up_from":346,"up_thru":372,"down_at":338,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2287832000},{"type":"v1","addr":"172.21.15.67:6818","nonce":2287832000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2287832000},{"type":"v1","addr":"172.21.15.67:6821","nonce":2287832000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2287832000},{"type":"v1","addr":"172.21.15.67:6828","nonce":2287832000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2287832000},{"type":"v1","addr":"172.21.15.67:6825","nonce":2287832000}]},"public_addr":"172.21.15.67:6818/2287832000","cluster_addr":"172.21.15.67:6821/2287832000","heartbeat_back_addr":"172.21.15.67:6828/2287832000","heartbeat_front_addr":"172.21.15.67:6825/2287832000","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":349,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":364,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6801","nonce":1504348724}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6803","nonce":1504348724}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6807","nonce":1504348724}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1504348724},{"type":"v1","addr":"172.21.15.67:6805","nonce":1504348724}]},"public_addr":"172.21.15.67:6801/1504348724","cluster_addr":"172.21.15.67:6803/1504348724","heartbeat_back_addr":"172.21.15.67:6807/1504348724","heartbeat_front_addr":"172.21.15.67:6805/1504348724","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":135,"last_clean_end":243,"up_from":250,"up_thru":372,"down_at":244,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6803","nonce":3986522908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6807","nonce":3986522908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6815","nonce":3986522908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6811","nonce":3986522908}]},"public_addr":"172.21.15.82:6803/3986522908","cluster_addr":"172.21.15.82:6807/3986522908","heartbeat_back_addr":"172.21.15.82:6815/3986522908","heartbeat_front_addr":"172.21.15.82:6811/3986522908","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":274,"last_clean_end":314,"up_from":320,"up_thru":372,"down_at":315,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":925466951},{"type":"v1","addr":"172.21.15.82:6817","nonce":925466951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":925466951},{"type":"v1","addr":"172.21.15.82:6819","nonce":925466951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":925466951},{"type":"v1","addr":"172.21.15.82:6823","nonce":925466951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":925466951},{"type":"v1","addr":"172.21.15.82:6821","nonce":925466951}]},"public_addr":"172.21.15.82:6817/925466951","cluster_addr":"172.21.15.82:6819/925466951","heartbeat_back_addr":"172.21.15.82:6823/925466951","heartbeat_front_addr":"172.21.15.82:6821/925466951","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":227,"last_clean_end":363,"up_from":372,"up_thru":372,"down_at":364,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6801","nonce":189641108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6806","nonce":189641108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6814","nonce":189641108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6810","nonce":189641108}]},"public_addr":"172.21.15.82:6801/189641108","cluster_addr":"172.21.15.82:6806/189641108","heartbeat_back_addr":"172.21.15.82:6814/189641108","heartbeat_front_addr":"172.21.15.82:6810/189641108","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":206,"last_clean_end":292,"up_from":298,"up_thru":372,"down_at":293,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3894469331},{"type":"v1","addr":"172.21.15.82:6825","nonce":3894469331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3894469331},{"type":"v1","addr":"172.21.15.82:6827","nonce":3894469331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3894469331},{"type":"v1","addr":"172.21.15.82:6831","nonce":3894469331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3894469331},{"type":"v1","addr":"172.21.15.82:6829","nonce":3894469331}]},"public_addr":"172.21.15.82:6825/3894469331","cluster_addr":"172.21.15.82:6827/3894469331","heartbeat_back_addr":"172.21.15.82:6831/3894469331","heartbeat_front_addr":"172.21.15.82:6829/3894469331","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:09:16.074169+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:08:34.393279+0000","dead_epoch":338},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:09:13.332707+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:08:29.898816+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:09:29.251348+0000","dead_epoch":244},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:09:27.308480+0000","dead_epoch":315},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:09:25.835127+0000","dead_epoch":364},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:09:31.013353+0000","dead_epoch":293}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:09:43.312 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:09:43.312 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:09:43.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:43.427+0000 7fc497bd3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:43.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:43.447+0000 7fc497bd3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:43.618 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:09:43.638 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:09:43.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:43.759+0000 7fc2f50e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:43.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:43.775+0000 7fc2f50e2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:43.956 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:09:43.956 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:09:43.975 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:09:43.975 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:09:43.975 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:09:43.975 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:09:43.975 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 3, 4, 7, 5, 1, 6] 2024-03-30T13:09:43.975 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:09:43.976 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:09:44.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:44.087+0000 7fbd073a1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:44.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:44.099+0000 7fbd073a1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:44.265 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:09:44.266 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:09:44.282 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":807,"stamp":"2024-03-30T13:09:44.025763+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":227698134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":391720,"kb_used_data":131060,"kb_used_omap":42,"kb_used_meta":260501,"kb_avail":754583000,"statfs":{"total":773094113280,"available":772692992000,"internally_reserved":0,"allocated":134205440,"data_stored":113569968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43263,"internal_metadata":266753793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.746929"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":857,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-30T13:09:40.530809+0000","last_change":"2024-03-30T13:09:27.171258+0000","last_active":"2024-03-30T13:09:40.530809+0000","last_peered":"2024-03-30T13:09:40.530809+0000","last_clean":"2024-03-30T13:09:40.530809+0000","last_became_active":"2024-03-30T13:09:27.168768+0000","last_became_peered":"2024-03-30T13:09:27.168768+0000","last_unstale":"2024-03-30T13:09:40.530809+0000","last_undegraded":"2024-03-30T13:09:40.530809+0000","last_fullsized":"2024-03-30T13:09:40.530809+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:08:59.481960+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:08:59.481960+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:13.508213+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":826,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-30T13:09:41.318922+0000","last_change":"2024-03-30T13:09:39.429119+0000","last_active":"2024-03-30T13:09:41.318922+0000","last_peered":"2024-03-30T13:09:41.318922+0000","last_clean":"2024-03-30T13:09:41.318922+0000","last_became_active":"2024-03-30T13:06:54.153354+0000","last_became_peered":"2024-03-30T13:06:54.153354+0000","last_unstale":"2024-03-30T13:09:41.318922+0000","last_undegraded":"2024-03-30T13:09:41.318922+0000","last_fullsized":"2024-03-30T13:09:41.318922+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:39.428863+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:09:39.428863+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:08.015563+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":454,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-30T13:09:41.682443+0000","last_change":"2024-03-30T13:08:39.542911+0000","last_active":"2024-03-30T13:09:41.682443+0000","last_peered":"2024-03-30T13:09:41.682443+0000","last_clean":"2024-03-30T13:09:41.682443+0000","last_became_active":"2024-03-30T13:08:39.541896+0000","last_became_peered":"2024-03-30T13:08:39.541896+0000","last_unstale":"2024-03-30T13:09:41.682443+0000","last_undegraded":"2024-03-30T13:09:41.682443+0000","last_fullsized":"2024-03-30T13:09:41.682443+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:08:37.747009+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:43.320096+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":424,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-30T13:09:41.681655+0000","last_change":"2024-03-30T13:09:27.165474+0000","last_active":"2024-03-30T13:09:41.681655+0000","last_peered":"2024-03-30T13:09:41.681655+0000","last_clean":"2024-03-30T13:09:41.681655+0000","last_became_active":"2024-03-30T13:09:27.164268+0000","last_became_peered":"2024-03-30T13:09:27.164268+0000","last_unstale":"2024-03-30T13:09:41.681655+0000","last_undegraded":"2024-03-30T13:09:41.681655+0000","last_fullsized":"2024-03-30T13:09:41.681655+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:08:47.761560+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:08:47.761560+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:48.582641+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":193,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-30T13:09:37.629370+0000","last_change":"2024-03-30T13:08:53.435394+0000","last_active":"2024-03-30T13:09:37.629370+0000","last_peered":"2024-03-30T13:09:37.629370+0000","last_clean":"2024-03-30T13:09:37.629370+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:09:37.629370+0000","last_undegraded":"2024-03-30T13:09:37.629370+0000","last_fullsized":"2024-03-30T13:09:37.629370+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:08:53.435125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:08:53.435125+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:55.081351+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":913,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-30T13:09:41.683265+0000","last_change":"2024-03-30T13:09:01.452554+0000","last_active":"2024-03-30T13:09:41.683265+0000","last_peered":"2024-03-30T13:09:41.683265+0000","last_clean":"2024-03-30T13:09:41.683265+0000","last_became_active":"2024-03-30T13:07:45.156174+0000","last_became_peered":"2024-03-30T13:07:45.156174+0000","last_unstale":"2024-03-30T13:09:41.683265+0000","last_undegraded":"2024-03-30T13:09:41.683265+0000","last_fullsized":"2024-03-30T13:09:41.683265+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:09:01.452292+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_clean_scrub_stamp":"2024-03-30T13:09:01.452292+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:03.443879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":56974905,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":441,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-30T13:09:42.174964+0000","last_change":"2024-03-30T13:09:42.175053+0000","last_active":"2024-03-30T13:09:42.174964+0000","last_peered":"2024-03-30T13:09:42.174964+0000","last_clean":"2024-03-30T13:09:42.174964+0000","last_became_active":"2024-03-30T13:06:54.155351+0000","last_became_peered":"2024-03-30T13:06:54.155351+0000","last_unstale":"2024-03-30T13:09:42.174964+0000","last_undegraded":"2024-03-30T13:09:42.174964+0000","last_fullsized":"2024-03-30T13:09:42.174964+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:09:42.174783+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:09:42.174783+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:00.350292+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":158,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-30T13:09:39.284954+0000","last_change":"2024-03-30T13:09:27.547106+0000","last_active":"2024-03-30T13:09:39.284954+0000","last_peered":"2024-03-30T13:09:39.284954+0000","last_clean":"2024-03-30T13:09:39.284954+0000","last_became_active":"2024-03-30T13:09:27.546829+0000","last_became_peered":"2024-03-30T13:09:27.546829+0000","last_unstale":"2024-03-30T13:09:39.284954+0000","last_undegraded":"2024-03-30T13:09:39.284954+0000","last_fullsized":"2024-03-30T13:09:39.284954+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:08:59.517955+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:08:59.517955+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:21.998387+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":991,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-30T13:09:41.683331+0000","last_change":"2024-03-30T13:09:34.162066+0000","last_active":"2024-03-30T13:09:41.683331+0000","last_peered":"2024-03-30T13:09:41.683331+0000","last_clean":"2024-03-30T13:09:41.683331+0000","last_became_active":"2024-03-30T13:07:45.156128+0000","last_became_peered":"2024-03-30T13:07:45.156128+0000","last_unstale":"2024-03-30T13:09:41.683331+0000","last_undegraded":"2024-03-30T13:09:41.683331+0000","last_fullsized":"2024-03-30T13:09:41.683331+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:09:34.161806+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:08:29.495729+0000","last_clean_scrub_stamp":"2024-03-30T13:09:34.161806+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:39.085627+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":65983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":744,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-30T13:09:41.318709+0000","last_change":"2024-03-30T13:09:27.170807+0000","last_active":"2024-03-30T13:09:41.318709+0000","last_peered":"2024-03-30T13:09:41.318709+0000","last_clean":"2024-03-30T13:09:41.318709+0000","last_became_active":"2024-03-30T13:09:27.169549+0000","last_became_peered":"2024-03-30T13:09:27.169549+0000","last_unstale":"2024-03-30T13:09:41.318709+0000","last_undegraded":"2024-03-30T13:09:41.318709+0000","last_fullsized":"2024-03-30T13:09:41.318709+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:15.464410+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:09:15.464410+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:39.896795+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":864,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-30T13:09:40.530339+0000","last_change":"2024-03-30T13:09:27.168331+0000","last_active":"2024-03-30T13:09:40.530339+0000","last_peered":"2024-03-30T13:09:40.530339+0000","last_clean":"2024-03-30T13:09:40.530339+0000","last_became_active":"2024-03-30T13:09:27.167070+0000","last_became_peered":"2024-03-30T13:09:27.167070+0000","last_unstale":"2024-03-30T13:09:40.530339+0000","last_undegraded":"2024-03-30T13:09:40.530339+0000","last_fullsized":"2024-03-30T13:09:40.530339+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:08:49.546103+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:08:49.546103+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:53.421805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":899,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-30T13:09:40.530596+0000","last_change":"2024-03-30T13:09:27.170410+0000","last_active":"2024-03-30T13:09:40.530596+0000","last_peered":"2024-03-30T13:09:40.530596+0000","last_clean":"2024-03-30T13:09:40.530596+0000","last_became_active":"2024-03-30T13:09:27.168910+0000","last_became_peered":"2024-03-30T13:09:27.168910+0000","last_unstale":"2024-03-30T13:09:40.530596+0000","last_undegraded":"2024-03-30T13:09:40.530596+0000","last_fullsized":"2024-03-30T13:09:40.530596+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:08:41.532076+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:08:41.532076+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:01.657802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":209,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-30T13:09:37.629403+0000","last_change":"2024-03-30T13:09:00.517090+0000","last_active":"2024-03-30T13:09:37.629403+0000","last_peered":"2024-03-30T13:09:37.629403+0000","last_clean":"2024-03-30T13:09:37.629403+0000","last_became_active":"2024-03-30T13:07:45.154855+0000","last_became_peered":"2024-03-30T13:07:45.154855+0000","last_unstale":"2024-03-30T13:09:37.629403+0000","last_undegraded":"2024-03-30T13:09:37.629403+0000","last_fullsized":"2024-03-30T13:09:37.629403+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:09:00.516843+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:09:00.516843+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:15.153201+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":371,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-30T13:09:41.681791+0000","last_change":"2024-03-30T13:08:24.700616+0000","last_active":"2024-03-30T13:09:41.681791+0000","last_peered":"2024-03-30T13:09:41.681791+0000","last_clean":"2024-03-30T13:09:41.681791+0000","last_became_active":"2024-03-30T13:06:54.153682+0000","last_became_peered":"2024-03-30T13:06:54.153682+0000","last_unstale":"2024-03-30T13:09:41.681791+0000","last_undegraded":"2024-03-30T13:09:41.681791+0000","last_fullsized":"2024-03-30T13:09:41.681791+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:08:24.700361+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:08:24.700361+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:54.523068+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":239,"reported_epoch":383,"state":"active+clean","last_fresh":"2024-03-30T13:09:37.629624+0000","last_change":"2024-03-30T13:09:34.558439+0000","last_active":"2024-03-30T13:09:37.629624+0000","last_peered":"2024-03-30T13:09:37.629624+0000","last_clean":"2024-03-30T13:09:37.629624+0000","last_became_active":"2024-03-30T13:09:27.162655+0000","last_became_peered":"2024-03-30T13:09:27.162655+0000","last_unstale":"2024-03-30T13:09:37.629624+0000","last_undegraded":"2024-03-30T13:09:37.629624+0000","last_fullsized":"2024-03-30T13:09:37.629624+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:09:34.558164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:09:34.558164+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:45.791448+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":133,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-03-30T13:09:39.285082+0000","last_change":"2024-03-30T13:08:52.434273+0000","last_active":"2024-03-30T13:09:39.285082+0000","last_peered":"2024-03-30T13:09:39.285082+0000","last_clean":"2024-03-30T13:09:39.285082+0000","last_became_active":"2024-03-30T13:08:39.543465+0000","last_became_peered":"2024-03-30T13:08:39.543465+0000","last_unstale":"2024-03-30T13:09:39.285082+0000","last_undegraded":"2024-03-30T13:09:39.285082+0000","last_fullsized":"2024-03-30T13:09:39.285082+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:08:52.434054+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:08:52.434054+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:05.803298+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":227698134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834117,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51832,"kb_used_data":19692,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94320008,"statfs":{"total":96636764160,"available":96583688192,"internally_reserved":0,"allocated":20164608,"data_stored":17578373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534745,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56988,"kb_used_data":24400,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94314852,"statfs":{"total":96636764160,"available":96578408448,"internally_reserved":0,"allocated":24985600,"data_stored":22419768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":33350120},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824057,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48732,"kb_used_data":16140,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94323108,"statfs":{"total":96636764160,"available":96586862592,"internally_reserved":0,"allocated":16527360,"data_stored":13941125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509613,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42056,"kb_used_data":10872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329784,"statfs":{"total":96636764160,"available":96593698816,"internally_reserved":0,"allocated":11132928,"data_stored":8567096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":346,"seq":1486058684431,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42304,"kb_used_data":10160,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94329536,"statfs":{"total":96636764160,"available":96593444864,"internally_reserved":0,"allocated":10403840,"data_stored":7831207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6045,"internal_metadata":32893027},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509613,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49244,"kb_used_data":18060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322596,"statfs":{"total":96636764160,"available":96586338304,"internally_reserved":0,"allocated":18493440,"data_stored":15920807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":298,"seq":1279900254244,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52288,"kb_used_data":14640,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94319552,"statfs":{"total":96636764160,"available":96583221248,"internally_reserved":0,"allocated":14991360,"data_stored":12398324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10986,"internal_metadata":38524182},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834575022,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48276,"kb_used_data":17096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323564,"statfs":{"total":96636764160,"available":96587329536,"internally_reserved":0,"allocated":17506304,"data_stored":14913268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:09:44.283 INFO:tasks.thrashosds.thrasher:Choosing PG 1.4 with acting set [3, 5, 7] 2024-03-30T13:09:44.283 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 2, 3, 4, 7, 5, 1, 6] 2024-03-30T13:09:44.284 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-30T13:09:44.284 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:09:44.374 INFO:tasks.ceph.osd.3:Stopped 2024-03-30T13:09:44.374 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 2, 3, 4, 7, 5, 1, 6] 2024-03-30T13:09:44.374 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 3 2024-03-30T13:09:44.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:44.491+0000 7f8205040640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:44.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:44.507+0000 7f8205040640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:44.687 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:09:44.687+0000 7fbb38f08640 -1 mon.a@0(leader).osd e388 definitely_dead 0 2024-03-30T13:09:45.345 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.3. 2024-03-30T13:09:45.370 INFO:tasks.thrashosds.thrasher:dead_osds=[3], live_osds=[0, 2, 4, 7, 5, 1, 6] 2024-03-30T13:09:45.371 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:09:45.371 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:09:45.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:45.479+0000 7f87e9c01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:45.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:09:45.491+0000 7f87e9c01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:09:45.660 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:09:45.661 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:09:45.677 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":808,"stamp":"2024-03-30T13:09:45.349385+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":227698134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":343660,"kb_used_data":114180,"kb_used_omap":40,"kb_used_meta":229335,"kb_avail":660259220,"statfs":{"total":676457349120,"available":676105441280,"internally_reserved":0,"allocated":116920320,"data_stored":98755466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41670,"internal_metadata":234839354},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.324966"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":857,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-30T13:09:40.530809+0000","last_change":"2024-03-30T13:09:27.171258+0000","last_active":"2024-03-30T13:09:40.530809+0000","last_peered":"2024-03-30T13:09:40.530809+0000","last_clean":"2024-03-30T13:09:40.530809+0000","last_became_active":"2024-03-30T13:09:27.168768+0000","last_became_peered":"2024-03-30T13:09:27.168768+0000","last_unstale":"2024-03-30T13:09:40.530809+0000","last_undegraded":"2024-03-30T13:09:40.530809+0000","last_fullsized":"2024-03-30T13:09:40.530809+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:08:59.481960+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:08:59.481960+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:13.508213+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":826,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-30T13:09:41.318922+0000","last_change":"2024-03-30T13:09:39.429119+0000","last_active":"2024-03-30T13:09:41.318922+0000","last_peered":"2024-03-30T13:09:41.318922+0000","last_clean":"2024-03-30T13:09:41.318922+0000","last_became_active":"2024-03-30T13:06:54.153354+0000","last_became_peered":"2024-03-30T13:06:54.153354+0000","last_unstale":"2024-03-30T13:09:41.318922+0000","last_undegraded":"2024-03-30T13:09:41.318922+0000","last_fullsized":"2024-03-30T13:09:41.318922+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:39.428863+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:09:39.428863+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:08.015563+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":454,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-30T13:09:41.682443+0000","last_change":"2024-03-30T13:08:39.542911+0000","last_active":"2024-03-30T13:09:41.682443+0000","last_peered":"2024-03-30T13:09:41.682443+0000","last_clean":"2024-03-30T13:09:41.682443+0000","last_became_active":"2024-03-30T13:08:39.541896+0000","last_became_peered":"2024-03-30T13:08:39.541896+0000","last_unstale":"2024-03-30T13:09:41.682443+0000","last_undegraded":"2024-03-30T13:09:41.682443+0000","last_fullsized":"2024-03-30T13:09:41.682443+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:08:37.747009+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:43.320096+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":424,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-30T13:09:41.681655+0000","last_change":"2024-03-30T13:09:27.165474+0000","last_active":"2024-03-30T13:09:41.681655+0000","last_peered":"2024-03-30T13:09:41.681655+0000","last_clean":"2024-03-30T13:09:41.681655+0000","last_became_active":"2024-03-30T13:09:27.164268+0000","last_became_peered":"2024-03-30T13:09:27.164268+0000","last_unstale":"2024-03-30T13:09:41.681655+0000","last_undegraded":"2024-03-30T13:09:41.681655+0000","last_fullsized":"2024-03-30T13:09:41.681655+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:08:47.761560+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T12:57:51.514144+0000","last_clean_scrub_stamp":"2024-03-30T13:08:47.761560+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:48.582641+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":202,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-30T13:09:43.388734+0000","last_change":"2024-03-30T13:08:53.435394+0000","last_active":"2024-03-30T13:09:43.388734+0000","last_peered":"2024-03-30T13:09:43.388734+0000","last_clean":"2024-03-30T13:09:43.388734+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:09:43.388734+0000","last_undegraded":"2024-03-30T13:09:43.388734+0000","last_fullsized":"2024-03-30T13:09:43.388734+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:08:53.435125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:08:53.435125+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:55.081351+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":913,"reported_epoch":387,"state":"stale+active+clean","last_fresh":"2024-03-30T13:09:41.683265+0000","last_change":"2024-03-30T13:09:01.452554+0000","last_active":"2024-03-30T13:09:41.683265+0000","last_peered":"2024-03-30T13:09:41.683265+0000","last_clean":"2024-03-30T13:09:41.683265+0000","last_became_active":"2024-03-30T13:07:45.156174+0000","last_became_peered":"2024-03-30T13:07:45.156174+0000","last_unstale":"2024-03-30T13:09:45.349417+0000","last_undegraded":"2024-03-30T13:09:41.683265+0000","last_fullsized":"2024-03-30T13:09:41.683265+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:09:01.452292+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_clean_scrub_stamp":"2024-03-30T13:09:01.452292+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:03.443879+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":56974905,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":441,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-30T13:09:42.174964+0000","last_change":"2024-03-30T13:09:42.175053+0000","last_active":"2024-03-30T13:09:42.174964+0000","last_peered":"2024-03-30T13:09:42.174964+0000","last_clean":"2024-03-30T13:09:42.174964+0000","last_became_active":"2024-03-30T13:06:54.155351+0000","last_became_peered":"2024-03-30T13:06:54.155351+0000","last_unstale":"2024-03-30T13:09:42.174964+0000","last_undegraded":"2024-03-30T13:09:42.174964+0000","last_fullsized":"2024-03-30T13:09:42.174964+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:09:42.174783+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:09:42.174783+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:00.350292+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":164,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-30T13:09:42.322613+0000","last_change":"2024-03-30T13:09:27.547106+0000","last_active":"2024-03-30T13:09:42.322613+0000","last_peered":"2024-03-30T13:09:42.322613+0000","last_clean":"2024-03-30T13:09:42.322613+0000","last_became_active":"2024-03-30T13:09:27.546829+0000","last_became_peered":"2024-03-30T13:09:27.546829+0000","last_unstale":"2024-03-30T13:09:42.322613+0000","last_undegraded":"2024-03-30T13:09:42.322613+0000","last_fullsized":"2024-03-30T13:09:42.322613+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:08:59.517955+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:08:59.517955+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:21.998387+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":991,"reported_epoch":387,"state":"stale+active+clean","last_fresh":"2024-03-30T13:09:41.683331+0000","last_change":"2024-03-30T13:09:34.162066+0000","last_active":"2024-03-30T13:09:41.683331+0000","last_peered":"2024-03-30T13:09:41.683331+0000","last_clean":"2024-03-30T13:09:41.683331+0000","last_became_active":"2024-03-30T13:07:45.156128+0000","last_became_peered":"2024-03-30T13:07:45.156128+0000","last_unstale":"2024-03-30T13:09:45.349422+0000","last_undegraded":"2024-03-30T13:09:41.683331+0000","last_fullsized":"2024-03-30T13:09:41.683331+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:09:34.161806+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:08:29.495729+0000","last_clean_scrub_stamp":"2024-03-30T13:09:34.161806+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:39.085627+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":65983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":744,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-30T13:09:41.318709+0000","last_change":"2024-03-30T13:09:27.170807+0000","last_active":"2024-03-30T13:09:41.318709+0000","last_peered":"2024-03-30T13:09:41.318709+0000","last_clean":"2024-03-30T13:09:41.318709+0000","last_became_active":"2024-03-30T13:09:27.169549+0000","last_became_peered":"2024-03-30T13:09:27.169549+0000","last_unstale":"2024-03-30T13:09:41.318709+0000","last_undegraded":"2024-03-30T13:09:41.318709+0000","last_fullsized":"2024-03-30T13:09:41.318709+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:15.464410+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:09:15.464410+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:39.896795+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":864,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-30T13:09:40.530339+0000","last_change":"2024-03-30T13:09:27.168331+0000","last_active":"2024-03-30T13:09:40.530339+0000","last_peered":"2024-03-30T13:09:40.530339+0000","last_clean":"2024-03-30T13:09:40.530339+0000","last_became_active":"2024-03-30T13:09:27.167070+0000","last_became_peered":"2024-03-30T13:09:27.167070+0000","last_unstale":"2024-03-30T13:09:40.530339+0000","last_undegraded":"2024-03-30T13:09:40.530339+0000","last_fullsized":"2024-03-30T13:09:40.530339+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:08:49.546103+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:08:49.546103+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:53.421805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":899,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-03-30T13:09:40.530596+0000","last_change":"2024-03-30T13:09:27.170410+0000","last_active":"2024-03-30T13:09:40.530596+0000","last_peered":"2024-03-30T13:09:40.530596+0000","last_clean":"2024-03-30T13:09:40.530596+0000","last_became_active":"2024-03-30T13:09:27.168910+0000","last_became_peered":"2024-03-30T13:09:27.168910+0000","last_unstale":"2024-03-30T13:09:40.530596+0000","last_undegraded":"2024-03-30T13:09:40.530596+0000","last_fullsized":"2024-03-30T13:09:40.530596+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:08:41.532076+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:03:28.655753+0000","last_clean_scrub_stamp":"2024-03-30T13:08:41.532076+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:01.657802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":218,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-30T13:09:43.388728+0000","last_change":"2024-03-30T13:09:00.517090+0000","last_active":"2024-03-30T13:09:43.388728+0000","last_peered":"2024-03-30T13:09:43.388728+0000","last_clean":"2024-03-30T13:09:43.388728+0000","last_became_active":"2024-03-30T13:07:45.154855+0000","last_became_peered":"2024-03-30T13:07:45.154855+0000","last_unstale":"2024-03-30T13:09:43.388728+0000","last_undegraded":"2024-03-30T13:09:43.388728+0000","last_fullsized":"2024-03-30T13:09:43.388728+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:09:00.516843+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:09:00.516843+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:15.153201+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":371,"reported_epoch":387,"state":"active+clean","last_fresh":"2024-03-30T13:09:41.681791+0000","last_change":"2024-03-30T13:08:24.700616+0000","last_active":"2024-03-30T13:09:41.681791+0000","last_peered":"2024-03-30T13:09:41.681791+0000","last_clean":"2024-03-30T13:09:41.681791+0000","last_became_active":"2024-03-30T13:06:54.153682+0000","last_became_peered":"2024-03-30T13:06:54.153682+0000","last_unstale":"2024-03-30T13:09:41.681791+0000","last_undegraded":"2024-03-30T13:09:41.681791+0000","last_fullsized":"2024-03-30T13:09:41.681791+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:08:24.700361+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:08:24.700361+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:09:54.523068+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":248,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-30T13:09:43.388691+0000","last_change":"2024-03-30T13:09:34.558439+0000","last_active":"2024-03-30T13:09:43.388691+0000","last_peered":"2024-03-30T13:09:43.388691+0000","last_clean":"2024-03-30T13:09:43.388691+0000","last_became_active":"2024-03-30T13:09:27.162655+0000","last_became_peered":"2024-03-30T13:09:27.162655+0000","last_unstale":"2024-03-30T13:09:43.388691+0000","last_undegraded":"2024-03-30T13:09:43.388691+0000","last_fullsized":"2024-03-30T13:09:43.388691+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:09:34.558164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:09:34.558164+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:45.791448+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":139,"reported_epoch":388,"state":"active+clean","last_fresh":"2024-03-30T13:09:42.322625+0000","last_change":"2024-03-30T13:08:52.434273+0000","last_active":"2024-03-30T13:09:42.322625+0000","last_peered":"2024-03-30T13:09:42.322625+0000","last_clean":"2024-03-30T13:09:42.322625+0000","last_became_active":"2024-03-30T13:08:39.543465+0000","last_became_peered":"2024-03-30T13:08:39.543465+0000","last_unstale":"2024-03-30T13:09:42.322625+0000","last_undegraded":"2024-03-30T13:09:42.322625+0000","last_fullsized":"2024-03-30T13:09:42.322625+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:08:52.434054+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:08:52.434054+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:05.803298+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":85,"num_read_kb":35,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":227698134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834117,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51832,"kb_used_data":19692,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94320008,"statfs":{"total":96636764160,"available":96583688192,"internally_reserved":0,"allocated":20164608,"data_stored":17578373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534746,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57048,"kb_used_data":24460,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94314792,"statfs":{"total":96636764160,"available":96578347008,"internally_reserved":0,"allocated":25047040,"data_stored":22447203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":33350120},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824057,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48732,"kb_used_data":16140,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94323108,"statfs":{"total":96636764160,"available":96586862592,"internally_reserved":0,"allocated":16527360,"data_stored":13941125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509614,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42116,"kb_used_data":10932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329724,"statfs":{"total":96636764160,"available":96593637376,"internally_reserved":0,"allocated":11194368,"data_stored":8594531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":346,"seq":1486058684432,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42356,"kb_used_data":10208,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94329484,"statfs":{"total":96636764160,"available":96593391616,"internally_reserved":0,"allocated":10452992,"data_stored":7853155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6045,"internal_metadata":32893027},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509614,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49288,"kb_used_data":18108,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322552,"statfs":{"total":96636764160,"available":96586293248,"internally_reserved":0,"allocated":18542592,"data_stored":15942755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":298,"seq":1279900254244,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52288,"kb_used_data":14640,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94319552,"statfs":{"total":96636764160,"available":96583221248,"internally_reserved":0,"allocated":14991360,"data_stored":12398324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10986,"internal_metadata":38524182},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:09:45.678 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:10:10.679 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:10:10.680 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:10:10.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:10.807+0000 7fcc703f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:10.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:10.819+0000 7fcc703f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:10.993 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:10:10.993 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:10:11.012 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":823,"stamp":"2024-03-30T13:10:10.034069+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":205513224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":357288,"kb_used_data":127836,"kb_used_omap":40,"kb_used_meta":229335,"kb_avail":660245592,"statfs":{"total":676457349120,"available":676091486208,"internally_reserved":0,"allocated":130904064,"data_stored":112564134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":41670,"internal_metadata":234839354},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004017"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":867,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-03-30T13:09:53.521586+0000","last_change":"2024-03-30T13:09:27.171258+0000","last_active":"2024-03-30T13:09:53.521586+0000","last_peered":"2024-03-30T13:09:53.521586+0000","last_clean":"2024-03-30T13:09:53.521586+0000","last_became_active":"2024-03-30T13:09:27.168768+0000","last_became_peered":"2024-03-30T13:09:27.168768+0000","last_unstale":"2024-03-30T13:09:53.521586+0000","last_undegraded":"2024-03-30T13:09:53.521586+0000","last_fullsized":"2024-03-30T13:09:53.521586+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:08:59.481960+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:08:59.481960+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:13.508213+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":844,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-03-30T13:09:53.437369+0000","last_change":"2024-03-30T13:09:46.402980+0000","last_active":"2024-03-30T13:09:53.437369+0000","last_peered":"2024-03-30T13:09:53.437369+0000","last_clean":"2024-03-30T13:09:53.437369+0000","last_became_active":"2024-03-30T13:09:46.368113+0000","last_became_peered":"2024-03-30T13:09:46.368113+0000","last_unstale":"2024-03-30T13:09:53.437369+0000","last_undegraded":"2024-03-30T13:09:53.437369+0000","last_fullsized":"2024-03-30T13:09:53.437369+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:39.428863+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:09:39.428863+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:08.721176+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":470,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-03-30T13:09:54.924751+0000","last_change":"2024-03-30T13:09:45.261607+0000","last_active":"2024-03-30T13:09:54.924751+0000","last_peered":"2024-03-30T13:09:54.924751+0000","last_clean":"2024-03-30T13:09:54.924751+0000","last_became_active":"2024-03-30T13:08:39.541896+0000","last_became_peered":"2024-03-30T13:08:39.541896+0000","last_unstale":"2024-03-30T13:09:54.924751+0000","last_undegraded":"2024-03-30T13:09:54.924751+0000","last_fullsized":"2024-03-30T13:09:54.924751+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:09:45.261353+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:09:45.261353+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:55.313728+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":440,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-03-30T13:09:54.924667+0000","last_change":"2024-03-30T13:09:53.320247+0000","last_active":"2024-03-30T13:09:54.924667+0000","last_peered":"2024-03-30T13:09:54.924667+0000","last_clean":"2024-03-30T13:09:54.924667+0000","last_became_active":"2024-03-30T13:09:27.164268+0000","last_became_peered":"2024-03-30T13:09:27.164268+0000","last_unstale":"2024-03-30T13:09:54.924667+0000","last_undegraded":"2024-03-30T13:09:54.924667+0000","last_fullsized":"2024-03-30T13:09:54.924667+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:09:53.319978+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:02.568868+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":217,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-03-30T13:09:56.639268+0000","last_change":"2024-03-30T13:09:56.639400+0000","last_active":"2024-03-30T13:09:56.639268+0000","last_peered":"2024-03-30T13:09:56.639268+0000","last_clean":"2024-03-30T13:09:56.639268+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:09:56.639268+0000","last_undegraded":"2024-03-30T13:09:56.639268+0000","last_fullsized":"2024-03-30T13:09:56.639268+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:09:56.639030+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:59.738230+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":25,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-03-30T13:09:53.521355+0000","last_change":"2024-03-30T13:09:46.467591+0000","last_active":"2024-03-30T13:09:53.521355+0000","last_peered":"2024-03-30T13:09:53.521355+0000","last_clean":"2024-03-30T13:09:53.521355+0000","last_became_active":"2024-03-30T13:09:46.379135+0000","last_became_peered":"2024-03-30T13:09:46.379135+0000","last_unstale":"2024-03-30T13:09:53.521355+0000","last_undegraded":"2024-03-30T13:09:53.521355+0000","last_fullsized":"2024-03-30T13:09:53.521355+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:09:01.452292+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_clean_scrub_stamp":"2024-03-30T13:09:01.452292+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:07.444644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11394981,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":449,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-03-30T13:09:54.924757+0000","last_change":"2024-03-30T13:09:42.175053+0000","last_active":"2024-03-30T13:09:54.924757+0000","last_peered":"2024-03-30T13:09:54.924757+0000","last_clean":"2024-03-30T13:09:54.924757+0000","last_became_active":"2024-03-30T13:06:54.155351+0000","last_became_peered":"2024-03-30T13:06:54.155351+0000","last_unstale":"2024-03-30T13:09:54.924757+0000","last_undegraded":"2024-03-30T13:09:54.924757+0000","last_fullsized":"2024-03-30T13:09:54.924757+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:09:42.174783+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:09:42.174783+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:00.350292+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":170,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-03-30T13:09:53.438527+0000","last_change":"2024-03-30T13:09:27.547106+0000","last_active":"2024-03-30T13:09:53.438527+0000","last_peered":"2024-03-30T13:09:53.438527+0000","last_clean":"2024-03-30T13:09:53.438527+0000","last_became_active":"2024-03-30T13:09:27.546829+0000","last_became_peered":"2024-03-30T13:09:27.546829+0000","last_unstale":"2024-03-30T13:09:53.438527+0000","last_undegraded":"2024-03-30T13:09:53.438527+0000","last_fullsized":"2024-03-30T13:09:53.438527+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:08:59.517955+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:08:59.517955+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:21.998387+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":19,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-03-30T13:09:53.521537+0000","last_change":"2024-03-30T13:09:47.144672+0000","last_active":"2024-03-30T13:09:53.521537+0000","last_peered":"2024-03-30T13:09:53.521537+0000","last_clean":"2024-03-30T13:09:53.521537+0000","last_became_active":"2024-03-30T13:09:46.375848+0000","last_became_peered":"2024-03-30T13:09:46.375848+0000","last_unstale":"2024-03-30T13:09:53.521537+0000","last_undegraded":"2024-03-30T13:09:53.521537+0000","last_fullsized":"2024-03-30T13:09:53.521537+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:09:34.161806+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:08:29.495729+0000","last_clean_scrub_stamp":"2024-03-30T13:09:34.161806+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:47.301833+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":55832128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":752,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-03-30T13:09:53.437438+0000","last_change":"2024-03-30T13:09:27.170807+0000","last_active":"2024-03-30T13:09:53.437438+0000","last_peered":"2024-03-30T13:09:53.437438+0000","last_clean":"2024-03-30T13:09:53.437438+0000","last_became_active":"2024-03-30T13:09:27.169549+0000","last_became_peered":"2024-03-30T13:09:27.169549+0000","last_unstale":"2024-03-30T13:09:53.437438+0000","last_undegraded":"2024-03-30T13:09:53.437438+0000","last_fullsized":"2024-03-30T13:09:53.437438+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:15.464410+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:09:15.464410+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:39.896795+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":882,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-03-30T13:09:54.854860+0000","last_change":"2024-03-30T13:09:54.854942+0000","last_active":"2024-03-30T13:09:54.854860+0000","last_peered":"2024-03-30T13:09:54.854860+0000","last_clean":"2024-03-30T13:09:54.854860+0000","last_became_active":"2024-03-30T13:09:27.167070+0000","last_became_peered":"2024-03-30T13:09:27.167070+0000","last_unstale":"2024-03-30T13:09:54.854860+0000","last_undegraded":"2024-03-30T13:09:54.854860+0000","last_fullsized":"2024-03-30T13:09:54.854860+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:09:54.854686+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:09:54.854686+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:19.248421+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":941,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-03-30T13:09:53.521573+0000","last_change":"2024-03-30T13:09:47.821350+0000","last_active":"2024-03-30T13:09:53.521573+0000","last_peered":"2024-03-30T13:09:53.521573+0000","last_clean":"2024-03-30T13:09:53.521573+0000","last_became_active":"2024-03-30T13:09:46.381369+0000","last_became_peered":"2024-03-30T13:09:46.381369+0000","last_unstale":"2024-03-30T13:09:53.521573+0000","last_undegraded":"2024-03-30T13:09:53.521573+0000","last_fullsized":"2024-03-30T13:09:53.521573+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:09:47.821071+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:16.112103+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":241,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-03-30T13:09:54.591871+0000","last_change":"2024-03-30T13:09:46.851385+0000","last_active":"2024-03-30T13:09:54.591871+0000","last_peered":"2024-03-30T13:09:54.591871+0000","last_clean":"2024-03-30T13:09:54.591871+0000","last_became_active":"2024-03-30T13:09:46.380997+0000","last_became_peered":"2024-03-30T13:09:46.380997+0000","last_unstale":"2024-03-30T13:09:54.591871+0000","last_undegraded":"2024-03-30T13:09:54.591871+0000","last_fullsized":"2024-03-30T13:09:54.591871+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:09:00.516843+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:09:00.516843+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:10.189819+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":397,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-03-30T13:09:54.924578+0000","last_change":"2024-03-30T13:09:50.334579+0000","last_active":"2024-03-30T13:09:54.924578+0000","last_peered":"2024-03-30T13:09:54.924578+0000","last_clean":"2024-03-30T13:09:54.924578+0000","last_became_active":"2024-03-30T13:09:46.374829+0000","last_became_peered":"2024-03-30T13:09:46.374829+0000","last_unstale":"2024-03-30T13:09:54.924578+0000","last_undegraded":"2024-03-30T13:09:54.924578+0000","last_fullsized":"2024-03-30T13:09:54.924578+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:50.334220+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:09:50.334220+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:02.033095+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":255,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-03-30T13:09:54.592122+0000","last_change":"2024-03-30T13:09:34.558439+0000","last_active":"2024-03-30T13:09:54.592122+0000","last_peered":"2024-03-30T13:09:54.592122+0000","last_clean":"2024-03-30T13:09:54.592122+0000","last_became_active":"2024-03-30T13:09:27.162655+0000","last_became_peered":"2024-03-30T13:09:27.162655+0000","last_unstale":"2024-03-30T13:09:54.592122+0000","last_undegraded":"2024-03-30T13:09:54.592122+0000","last_fullsized":"2024-03-30T13:09:54.592122+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:09:34.558164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:09:34.558164+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:45.791448+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":153,"reported_epoch":391,"state":"active+clean","last_fresh":"2024-03-30T13:10:08.609517+0000","last_change":"2024-03-30T13:10:08.609586+0000","last_active":"2024-03-30T13:10:08.609517+0000","last_peered":"2024-03-30T13:10:08.609517+0000","last_clean":"2024-03-30T13:10:08.609517+0000","last_became_active":"2024-03-30T13:08:39.543465+0000","last_became_peered":"2024-03-30T13:08:39.543465+0000","last_unstale":"2024-03-30T13:10:08.609517+0000","last_undegraded":"2024-03-30T13:10:08.609517+0000","last_fullsized":"2024-03-30T13:10:08.609517+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:10:08.609336+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:10:08.609336+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:17.880548+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":205513224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":110415872,"data_stored":110415872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834122,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57008,"kb_used_data":24868,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94314832,"statfs":{"total":96636764160,"available":96578387968,"internally_reserved":0,"allocated":25464832,"data_stored":22844842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534751,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57080,"kb_used_data":24496,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94314760,"statfs":{"total":96636764160,"available":96578314240,"internally_reserved":0,"allocated":25083904,"data_stored":22463914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":33350120},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824062,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48788,"kb_used_data":16200,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94323052,"statfs":{"total":96636764160,"available":96586805248,"internally_reserved":0,"allocated":16588800,"data_stored":13968810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509619,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42148,"kb_used_data":10968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329692,"statfs":{"total":96636764160,"available":96593604608,"internally_reserved":0,"allocated":11231232,"data_stored":8611242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":346,"seq":1486058684437,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43044,"kb_used_data":10904,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94328796,"statfs":{"total":96636764160,"available":96592687104,"internally_reserved":0,"allocated":11165696,"data_stored":8545706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6045,"internal_metadata":32893027},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509619,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56888,"kb_used_data":25712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94314952,"statfs":{"total":96636764160,"available":96578510848,"internally_reserved":0,"allocated":26329088,"data_stored":23709098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":298,"seq":1279900254249,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52332,"kb_used_data":14688,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94319508,"statfs":{"total":96636764160,"available":96583176192,"internally_reserved":0,"allocated":15040512,"data_stored":12420522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10986,"internal_metadata":38524182},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21573632,"data_stored":21573632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6410240,"data_stored":6410240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20709376,"data_stored":20709376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:10:11.013 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:10:11.013 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:10:11.013 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-03-30T13:10:11.013 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-30T13:10:11.013 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-30T13:10:11.016 INFO:tasks.ceph.osd.3:Started 2024-03-30T13:10:11.016 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-30T13:10:11.077 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T13:10:11.075+0000 7ffb6c4cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:11.084 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T13:10:11.083+0000 7ffb6c4cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:11.084 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T13:10:11.083+0000 7ffb6c4cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:11.126 INFO:teuthology.orchestra.run.smithi067.stderr:no valid command found; 10 closest matches: 2024-03-30T13:10:11.127 INFO:teuthology.orchestra.run.smithi067.stderr:0 2024-03-30T13:10:11.127 INFO:teuthology.orchestra.run.smithi067.stderr:1 2024-03-30T13:10:11.127 INFO:teuthology.orchestra.run.smithi067.stderr:2 2024-03-30T13:10:11.127 INFO:teuthology.orchestra.run.smithi067.stderr:abort 2024-03-30T13:10:11.127 INFO:teuthology.orchestra.run.smithi067.stderr:assert 2024-03-30T13:10:11.127 INFO:teuthology.orchestra.run.smithi067.stderr:config diff 2024-03-30T13:10:11.127 INFO:teuthology.orchestra.run.smithi067.stderr:config diff get 2024-03-30T13:10:11.127 INFO:teuthology.orchestra.run.smithi067.stderr:config get 2024-03-30T13:10:11.127 INFO:teuthology.orchestra.run.smithi067.stderr:config help [] 2024-03-30T13:10:11.127 INFO:teuthology.orchestra.run.smithi067.stderr:config set ... 2024-03-30T13:10:11.127 INFO:teuthology.orchestra.run.smithi067.stderr:admin_socket: invalid command 2024-03-30T13:10:11.128 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:10:11.129 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-03-30T13:10:11.616 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T13:10:11.615+0000 7ffb6c4cd7c0 -1 Falling back to public interface 2024-03-30T13:10:15.365 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T13:10:15.363+0000 7ffb6c4cd7c0 -1 osd.3 388 log_to_monitors true 2024-03-30T13:10:16.130 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-30T13:10:16.190 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T13:10:16.190+0000 7ffb642a2640 -1 osd.3 388 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:10:16.264 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-30T13:10:16.264 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-30T13:10:16.264 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-03-30T13:10:16.264 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-30T13:10:16.279 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:10:16.279 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-03-30T13:10:16.279 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 3 2024-03-30T13:10:16.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:16.398+0000 7f4c2e94c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:16.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:16.410+0000 7f4c2e94c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:16.593 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:10:16.590+0000 7fbb38f08640 -1 mon.a@0(leader).osd e392 definitely_dead 0 2024-03-30T13:10:17.148 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.3. 2024-03-30T13:10:17.181 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-03-30T13:10:17.181 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:10:17.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:17.294+0000 7f8c628e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:17.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:17.310+0000 7f8c628e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:18.161 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:10:18.191 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:10:18.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:18.306+0000 7ff45b9ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:18.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:18.322+0000 7ff45b9ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:19.165 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:10:19.191 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:10:19.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:19.310+0000 7f5e20b20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:19.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:19.322+0000 7f5e20b20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:20.183 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:10:20.206 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:10:20.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:20.326+0000 7f474fef7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:20.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:20.342+0000 7f474fef7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:21.179 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:10:21.208 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:10:21.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:21.326+0000 7f6990409640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:21.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:21.346+0000 7f6990409640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:22.199 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:10:22.227 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:10:22.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:22.350+0000 7f56d845d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:22.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:22.366+0000 7f56d845d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:23.207 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:10:23.233 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:10:23.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:23.350+0000 7f2fca480640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:23.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:23.366+0000 7f2fca480640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:24.212 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:10:24.240 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:10:24.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:24.358+0000 7f74ed1ed640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:24.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:24.378+0000 7f74ed1ed640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:25.231 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:10:25.255 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:10:25.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:25.378+0000 7f09854df640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:25.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:25.394+0000 7f09854df640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:26.240 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:10:26.266 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:10:26.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:26.390+0000 7f50d27bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:26.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:26.406+0000 7f50d27bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:27.252 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:10:27.275 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:10:27.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:27.398+0000 7f008e985640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:27.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:27.414+0000 7f008e985640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:28.263 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:10:28.293 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:10:28.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:28.418+0000 7fa14eecb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:28.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:28.434+0000 7fa14eecb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:29.274 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:10:29.307 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:10:29.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:29.430+0000 7f233863e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:29.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:29.446+0000 7f233863e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:30.288 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:10:30.318 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:10:30.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:30.442+0000 7fe798618640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:30.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:30.462+0000 7fe798618640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:31.298 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:10:31.326 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:10:31.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:31.450+0000 7f76d63d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:31.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:31.466+0000 7f76d63d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:32.328 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:10:32.353 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:10:32.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:32.478+0000 7f8d87133640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:32.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:32.494+0000 7f8d87133640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:33.325 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:10:33.350 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:10:33.351 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:10:33.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:33.470+0000 7fa1ca4cd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:33.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:33.486+0000 7fa1ca4cd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:33.657 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:10:33.658 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:10:33.676 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":852,"stamp":"2024-03-30T13:10:33.329493+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":189042595,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":394184,"kb_used_data":133016,"kb_used_omap":44,"kb_used_meta":261011,"kb_avail":754580536,"statfs":{"total":773094113280,"available":772690468864,"internally_reserved":0,"allocated":136208384,"data_stored":114479349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":46068,"internal_metadata":267275276},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.051073"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":904,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-30T13:10:30.492174+0000","last_change":"2024-03-30T13:10:16.176071+0000","last_active":"2024-03-30T13:10:30.492174+0000","last_peered":"2024-03-30T13:10:30.492174+0000","last_clean":"2024-03-30T13:10:30.492174+0000","last_became_active":"2024-03-30T13:09:27.168768+0000","last_became_peered":"2024-03-30T13:09:27.168768+0000","last_unstale":"2024-03-30T13:10:30.492174+0000","last_undegraded":"2024-03-30T13:10:30.492174+0000","last_fullsized":"2024-03-30T13:10:30.492174+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:10:16.175659+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:10:16.175659+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:39.530667+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":879,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-30T13:10:30.295653+0000","last_change":"2024-03-30T13:10:18.178182+0000","last_active":"2024-03-30T13:10:30.295653+0000","last_peered":"2024-03-30T13:10:30.295653+0000","last_clean":"2024-03-30T13:10:30.295653+0000","last_became_active":"2024-03-30T13:10:18.176067+0000","last_became_peered":"2024-03-30T13:10:18.176067+0000","last_unstale":"2024-03-30T13:10:30.295653+0000","last_undegraded":"2024-03-30T13:10:30.295653+0000","last_fullsized":"2024-03-30T13:10:30.295653+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:39.428863+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:09:39.428863+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:05.213207+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":502,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-30T13:10:32.832046+0000","last_change":"2024-03-30T13:09:45.261607+0000","last_active":"2024-03-30T13:10:32.832046+0000","last_peered":"2024-03-30T13:10:32.832046+0000","last_clean":"2024-03-30T13:10:32.832046+0000","last_became_active":"2024-03-30T13:08:39.541896+0000","last_became_peered":"2024-03-30T13:08:39.541896+0000","last_unstale":"2024-03-30T13:10:32.832046+0000","last_undegraded":"2024-03-30T13:10:32.832046+0000","last_fullsized":"2024-03-30T13:10:32.832046+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:09:45.261353+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:09:45.261353+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:55.313728+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":472,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-30T13:10:32.831677+0000","last_change":"2024-03-30T13:09:53.320247+0000","last_active":"2024-03-30T13:10:32.831677+0000","last_peered":"2024-03-30T13:10:32.831677+0000","last_clean":"2024-03-30T13:10:32.831677+0000","last_became_active":"2024-03-30T13:09:27.164268+0000","last_became_peered":"2024-03-30T13:09:27.164268+0000","last_unstale":"2024-03-30T13:10:32.831677+0000","last_undegraded":"2024-03-30T13:10:32.831677+0000","last_fullsized":"2024-03-30T13:10:32.831677+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:09:53.319978+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:02.568868+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":242,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-03-30T13:10:28.272119+0000","last_change":"2024-03-30T13:09:56.639400+0000","last_active":"2024-03-30T13:10:28.272119+0000","last_peered":"2024-03-30T13:10:28.272119+0000","last_clean":"2024-03-30T13:10:28.272119+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:10:28.272119+0000","last_undegraded":"2024-03-30T13:10:28.272119+0000","last_fullsized":"2024-03-30T13:10:28.272119+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:09:56.639030+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:59.738230+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":910,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-03-30T13:10:31.307754+0000","last_change":"2024-03-30T13:10:18.181812+0000","last_active":"2024-03-30T13:10:31.307754+0000","last_peered":"2024-03-30T13:10:31.307754+0000","last_clean":"2024-03-30T13:10:31.307754+0000","last_became_active":"2024-03-30T13:10:18.181149+0000","last_became_peered":"2024-03-30T13:10:18.181149+0000","last_unstale":"2024-03-30T13:10:31.307754+0000","last_undegraded":"2024-03-30T13:10:31.307754+0000","last_fullsized":"2024-03-30T13:10:31.307754+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:10:09.027639+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_clean_scrub_stamp":"2024-03-30T13:10:09.027639+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:09.537539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":481,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-30T13:10:32.831802+0000","last_change":"2024-03-30T13:09:42.175053+0000","last_active":"2024-03-30T13:10:32.831802+0000","last_peered":"2024-03-30T13:10:32.831802+0000","last_clean":"2024-03-30T13:10:32.831802+0000","last_became_active":"2024-03-30T13:06:54.155351+0000","last_became_peered":"2024-03-30T13:06:54.155351+0000","last_unstale":"2024-03-30T13:10:32.831802+0000","last_undegraded":"2024-03-30T13:10:32.831802+0000","last_fullsized":"2024-03-30T13:10:32.831802+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:09:42.174783+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:09:42.174783+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:00.350292+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":206,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-30T13:10:29.283937+0000","last_change":"2024-03-30T13:10:23.571319+0000","last_active":"2024-03-30T13:10:29.283937+0000","last_peered":"2024-03-30T13:10:29.283937+0000","last_clean":"2024-03-30T13:10:29.283937+0000","last_became_active":"2024-03-30T13:09:27.546829+0000","last_became_peered":"2024-03-30T13:09:27.546829+0000","last_unstale":"2024-03-30T13:10:29.283937+0000","last_undegraded":"2024-03-30T13:10:29.283937+0000","last_fullsized":"2024-03-30T13:10:29.283937+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:10:23.570911+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:10:23.570911+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:45.997399+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1011,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-03-30T13:10:31.307649+0000","last_change":"2024-03-30T13:10:18.715230+0000","last_active":"2024-03-30T13:10:31.307649+0000","last_peered":"2024-03-30T13:10:31.307649+0000","last_clean":"2024-03-30T13:10:31.307649+0000","last_became_active":"2024-03-30T13:10:18.714034+0000","last_became_peered":"2024-03-30T13:10:18.714034+0000","last_unstale":"2024-03-30T13:10:31.307649+0000","last_undegraded":"2024-03-30T13:10:31.307649+0000","last_fullsized":"2024-03-30T13:10:31.307649+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:09:34.161806+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:08:29.495729+0000","last_clean_scrub_stamp":"2024-03-30T13:09:34.161806+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:43.489690+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":50756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":781,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-30T13:10:30.295379+0000","last_change":"2024-03-30T13:09:27.170807+0000","last_active":"2024-03-30T13:10:30.295379+0000","last_peered":"2024-03-30T13:10:30.295379+0000","last_clean":"2024-03-30T13:10:30.295379+0000","last_became_active":"2024-03-30T13:09:27.169549+0000","last_became_peered":"2024-03-30T13:09:27.169549+0000","last_unstale":"2024-03-30T13:10:30.295379+0000","last_undegraded":"2024-03-30T13:10:30.295379+0000","last_fullsized":"2024-03-30T13:10:30.295379+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:15.464410+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:09:15.464410+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:39.896795+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":911,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-30T13:10:30.491918+0000","last_change":"2024-03-30T13:09:54.854942+0000","last_active":"2024-03-30T13:10:30.491918+0000","last_peered":"2024-03-30T13:10:30.491918+0000","last_clean":"2024-03-30T13:10:30.491918+0000","last_became_active":"2024-03-30T13:09:27.167070+0000","last_became_peered":"2024-03-30T13:09:27.167070+0000","last_unstale":"2024-03-30T13:10:30.491918+0000","last_undegraded":"2024-03-30T13:10:30.491918+0000","last_fullsized":"2024-03-30T13:10:30.491918+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:09:54.854686+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:09:54.854686+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:19.248421+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":976,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-30T13:10:30.491749+0000","last_change":"2024-03-30T13:10:18.181749+0000","last_active":"2024-03-30T13:10:30.491749+0000","last_peered":"2024-03-30T13:10:30.491749+0000","last_clean":"2024-03-30T13:10:30.491749+0000","last_became_active":"2024-03-30T13:10:18.180670+0000","last_became_peered":"2024-03-30T13:10:18.180670+0000","last_unstale":"2024-03-30T13:10:30.491749+0000","last_undegraded":"2024-03-30T13:10:30.491749+0000","last_fullsized":"2024-03-30T13:10:30.491749+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:09:47.821071+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:11.230137+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":280,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-03-30T13:10:28.271819+0000","last_change":"2024-03-30T13:10:18.185077+0000","last_active":"2024-03-30T13:10:28.271819+0000","last_peered":"2024-03-30T13:10:28.271819+0000","last_clean":"2024-03-30T13:10:28.271819+0000","last_became_active":"2024-03-30T13:10:18.183814+0000","last_became_peered":"2024-03-30T13:10:18.183814+0000","last_unstale":"2024-03-30T13:10:28.271819+0000","last_undegraded":"2024-03-30T13:10:28.271819+0000","last_fullsized":"2024-03-30T13:10:28.271819+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:10:13.731448+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:10:13.731448+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:23.829826+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":435,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-30T13:10:32.831842+0000","last_change":"2024-03-30T13:10:18.175121+0000","last_active":"2024-03-30T13:10:32.831842+0000","last_peered":"2024-03-30T13:10:32.831842+0000","last_clean":"2024-03-30T13:10:32.831842+0000","last_became_active":"2024-03-30T13:10:18.174189+0000","last_became_peered":"2024-03-30T13:10:18.174189+0000","last_unstale":"2024-03-30T13:10:32.831842+0000","last_undegraded":"2024-03-30T13:10:32.831842+0000","last_fullsized":"2024-03-30T13:10:32.831842+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:50.334220+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:09:50.334220+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:14.996022+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":280,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-03-30T13:10:28.271839+0000","last_change":"2024-03-30T13:09:34.558439+0000","last_active":"2024-03-30T13:10:28.271839+0000","last_peered":"2024-03-30T13:10:28.271839+0000","last_clean":"2024-03-30T13:10:28.271839+0000","last_became_active":"2024-03-30T13:09:27.162655+0000","last_became_peered":"2024-03-30T13:09:27.162655+0000","last_unstale":"2024-03-30T13:10:28.271839+0000","last_undegraded":"2024-03-30T13:10:28.271839+0000","last_fullsized":"2024-03-30T13:10:28.271839+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:09:34.558164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:09:34.558164+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:45.791448+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":181,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-30T13:10:29.283948+0000","last_change":"2024-03-30T13:10:08.609586+0000","last_active":"2024-03-30T13:10:29.283948+0000","last_peered":"2024-03-30T13:10:29.283948+0000","last_clean":"2024-03-30T13:10:29.283948+0000","last_became_active":"2024-03-30T13:08:39.543465+0000","last_became_peered":"2024-03-30T13:08:39.543465+0000","last_unstale":"2024-03-30T13:10:29.283948+0000","last_undegraded":"2024-03-30T13:10:29.283948+0000","last_fullsized":"2024-03-30T13:10:29.283948+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:10:08.609336+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:10:08.609336+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:17.880548+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":189042595,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834127,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52076,"kb_used_data":19932,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94319764,"statfs":{"total":96636764160,"available":96583438336,"internally_reserved":0,"allocated":20410368,"data_stored":17689988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534755,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57240,"kb_used_data":24652,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94314600,"statfs":{"total":96636764160,"available":96578150400,"internally_reserved":0,"allocated":25243648,"data_stored":22536870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":33350120},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824067,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48968,"kb_used_data":16380,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94322872,"statfs":{"total":96636764160,"available":96586620928,"internally_reserved":0,"allocated":16773120,"data_stored":14052740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509623,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42308,"kb_used_data":11124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329532,"statfs":{"total":96636764160,"available":96593440768,"internally_reserved":0,"allocated":11390976,"data_stored":8684198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":346,"seq":1486058684441,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42540,"kb_used_data":10400,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94329300,"statfs":{"total":96636764160,"available":96593203200,"internally_reserved":0,"allocated":10649600,"data_stored":7942822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6045,"internal_metadata":32893027},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509623,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49496,"kb_used_data":18312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322344,"statfs":{"total":96636764160,"available":96586080256,"internally_reserved":0,"allocated":18751488,"data_stored":16037909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":298,"seq":1279900254254,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52524,"kb_used_data":14880,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94319316,"statfs":{"total":96636764160,"available":96582979584,"internally_reserved":0,"allocated":15237120,"data_stored":12509939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10986,"internal_metadata":38524182},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147333,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49032,"kb_used_data":17336,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94322808,"statfs":{"total":96636764160,"available":96586555392,"internally_reserved":0,"allocated":17752064,"data_stored":15024883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:10:33.678 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:10:33.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:33.798+0000 7f573671a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:33.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:33.810+0000 7f573671a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:10:33.985 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:10:34.003 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":852,"stamp":"2024-03-30T13:10:33.329493+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":189042595,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":394184,"kb_used_data":133016,"kb_used_omap":44,"kb_used_meta":261011,"kb_avail":754580536,"statfs":{"total":773094113280,"available":772690468864,"internally_reserved":0,"allocated":136208384,"data_stored":114479349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":46068,"internal_metadata":267275276},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.051073"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":904,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-30T13:10:30.492174+0000","last_change":"2024-03-30T13:10:16.176071+0000","last_active":"2024-03-30T13:10:30.492174+0000","last_peered":"2024-03-30T13:10:30.492174+0000","last_clean":"2024-03-30T13:10:30.492174+0000","last_became_active":"2024-03-30T13:09:27.168768+0000","last_became_peered":"2024-03-30T13:09:27.168768+0000","last_unstale":"2024-03-30T13:10:30.492174+0000","last_undegraded":"2024-03-30T13:10:30.492174+0000","last_fullsized":"2024-03-30T13:10:30.492174+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:10:16.175659+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:10:16.175659+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:39.530667+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":879,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-30T13:10:30.295653+0000","last_change":"2024-03-30T13:10:18.178182+0000","last_active":"2024-03-30T13:10:30.295653+0000","last_peered":"2024-03-30T13:10:30.295653+0000","last_clean":"2024-03-30T13:10:30.295653+0000","last_became_active":"2024-03-30T13:10:18.176067+0000","last_became_peered":"2024-03-30T13:10:18.176067+0000","last_unstale":"2024-03-30T13:10:30.295653+0000","last_undegraded":"2024-03-30T13:10:30.295653+0000","last_fullsized":"2024-03-30T13:10:30.295653+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:39.428863+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:09:39.428863+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:05.213207+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":502,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-30T13:10:32.832046+0000","last_change":"2024-03-30T13:09:45.261607+0000","last_active":"2024-03-30T13:10:32.832046+0000","last_peered":"2024-03-30T13:10:32.832046+0000","last_clean":"2024-03-30T13:10:32.832046+0000","last_became_active":"2024-03-30T13:08:39.541896+0000","last_became_peered":"2024-03-30T13:08:39.541896+0000","last_unstale":"2024-03-30T13:10:32.832046+0000","last_undegraded":"2024-03-30T13:10:32.832046+0000","last_fullsized":"2024-03-30T13:10:32.832046+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:09:45.261353+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:09:45.261353+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:55.313728+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":472,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-30T13:10:32.831677+0000","last_change":"2024-03-30T13:09:53.320247+0000","last_active":"2024-03-30T13:10:32.831677+0000","last_peered":"2024-03-30T13:10:32.831677+0000","last_clean":"2024-03-30T13:10:32.831677+0000","last_became_active":"2024-03-30T13:09:27.164268+0000","last_became_peered":"2024-03-30T13:09:27.164268+0000","last_unstale":"2024-03-30T13:10:32.831677+0000","last_undegraded":"2024-03-30T13:10:32.831677+0000","last_fullsized":"2024-03-30T13:10:32.831677+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:09:53.319978+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:02.568868+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":242,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-03-30T13:10:28.272119+0000","last_change":"2024-03-30T13:09:56.639400+0000","last_active":"2024-03-30T13:10:28.272119+0000","last_peered":"2024-03-30T13:10:28.272119+0000","last_clean":"2024-03-30T13:10:28.272119+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:10:28.272119+0000","last_undegraded":"2024-03-30T13:10:28.272119+0000","last_fullsized":"2024-03-30T13:10:28.272119+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:09:56.639030+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:59.738230+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":910,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-03-30T13:10:31.307754+0000","last_change":"2024-03-30T13:10:18.181812+0000","last_active":"2024-03-30T13:10:31.307754+0000","last_peered":"2024-03-30T13:10:31.307754+0000","last_clean":"2024-03-30T13:10:31.307754+0000","last_became_active":"2024-03-30T13:10:18.181149+0000","last_became_peered":"2024-03-30T13:10:18.181149+0000","last_unstale":"2024-03-30T13:10:31.307754+0000","last_undegraded":"2024-03-30T13:10:31.307754+0000","last_fullsized":"2024-03-30T13:10:31.307754+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:10:09.027639+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_clean_scrub_stamp":"2024-03-30T13:10:09.027639+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:09.537539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":481,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-30T13:10:32.831802+0000","last_change":"2024-03-30T13:09:42.175053+0000","last_active":"2024-03-30T13:10:32.831802+0000","last_peered":"2024-03-30T13:10:32.831802+0000","last_clean":"2024-03-30T13:10:32.831802+0000","last_became_active":"2024-03-30T13:06:54.155351+0000","last_became_peered":"2024-03-30T13:06:54.155351+0000","last_unstale":"2024-03-30T13:10:32.831802+0000","last_undegraded":"2024-03-30T13:10:32.831802+0000","last_fullsized":"2024-03-30T13:10:32.831802+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:09:42.174783+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:09:42.174783+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:00.350292+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":206,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-30T13:10:29.283937+0000","last_change":"2024-03-30T13:10:23.571319+0000","last_active":"2024-03-30T13:10:29.283937+0000","last_peered":"2024-03-30T13:10:29.283937+0000","last_clean":"2024-03-30T13:10:29.283937+0000","last_became_active":"2024-03-30T13:09:27.546829+0000","last_became_peered":"2024-03-30T13:09:27.546829+0000","last_unstale":"2024-03-30T13:10:29.283937+0000","last_undegraded":"2024-03-30T13:10:29.283937+0000","last_fullsized":"2024-03-30T13:10:29.283937+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:10:23.570911+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:10:23.570911+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:45.997399+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1011,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-03-30T13:10:31.307649+0000","last_change":"2024-03-30T13:10:18.715230+0000","last_active":"2024-03-30T13:10:31.307649+0000","last_peered":"2024-03-30T13:10:31.307649+0000","last_clean":"2024-03-30T13:10:31.307649+0000","last_became_active":"2024-03-30T13:10:18.714034+0000","last_became_peered":"2024-03-30T13:10:18.714034+0000","last_unstale":"2024-03-30T13:10:31.307649+0000","last_undegraded":"2024-03-30T13:10:31.307649+0000","last_fullsized":"2024-03-30T13:10:31.307649+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:09:34.161806+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:08:29.495729+0000","last_clean_scrub_stamp":"2024-03-30T13:09:34.161806+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:43.489690+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":50756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":781,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-30T13:10:30.295379+0000","last_change":"2024-03-30T13:09:27.170807+0000","last_active":"2024-03-30T13:10:30.295379+0000","last_peered":"2024-03-30T13:10:30.295379+0000","last_clean":"2024-03-30T13:10:30.295379+0000","last_became_active":"2024-03-30T13:09:27.169549+0000","last_became_peered":"2024-03-30T13:09:27.169549+0000","last_unstale":"2024-03-30T13:10:30.295379+0000","last_undegraded":"2024-03-30T13:10:30.295379+0000","last_fullsized":"2024-03-30T13:10:30.295379+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:15.464410+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:09:15.464410+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:39.896795+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":911,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-30T13:10:30.491918+0000","last_change":"2024-03-30T13:09:54.854942+0000","last_active":"2024-03-30T13:10:30.491918+0000","last_peered":"2024-03-30T13:10:30.491918+0000","last_clean":"2024-03-30T13:10:30.491918+0000","last_became_active":"2024-03-30T13:09:27.167070+0000","last_became_peered":"2024-03-30T13:09:27.167070+0000","last_unstale":"2024-03-30T13:10:30.491918+0000","last_undegraded":"2024-03-30T13:10:30.491918+0000","last_fullsized":"2024-03-30T13:10:30.491918+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:09:54.854686+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:09:54.854686+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:19.248421+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":976,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-30T13:10:30.491749+0000","last_change":"2024-03-30T13:10:18.181749+0000","last_active":"2024-03-30T13:10:30.491749+0000","last_peered":"2024-03-30T13:10:30.491749+0000","last_clean":"2024-03-30T13:10:30.491749+0000","last_became_active":"2024-03-30T13:10:18.180670+0000","last_became_peered":"2024-03-30T13:10:18.180670+0000","last_unstale":"2024-03-30T13:10:30.491749+0000","last_undegraded":"2024-03-30T13:10:30.491749+0000","last_fullsized":"2024-03-30T13:10:30.491749+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:09:47.821071+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:11.230137+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":280,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-03-30T13:10:28.271819+0000","last_change":"2024-03-30T13:10:18.185077+0000","last_active":"2024-03-30T13:10:28.271819+0000","last_peered":"2024-03-30T13:10:28.271819+0000","last_clean":"2024-03-30T13:10:28.271819+0000","last_became_active":"2024-03-30T13:10:18.183814+0000","last_became_peered":"2024-03-30T13:10:18.183814+0000","last_unstale":"2024-03-30T13:10:28.271819+0000","last_undegraded":"2024-03-30T13:10:28.271819+0000","last_fullsized":"2024-03-30T13:10:28.271819+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:10:13.731448+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:10:13.731448+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:23.829826+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":435,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-30T13:10:32.831842+0000","last_change":"2024-03-30T13:10:18.175121+0000","last_active":"2024-03-30T13:10:32.831842+0000","last_peered":"2024-03-30T13:10:32.831842+0000","last_clean":"2024-03-30T13:10:32.831842+0000","last_became_active":"2024-03-30T13:10:18.174189+0000","last_became_peered":"2024-03-30T13:10:18.174189+0000","last_unstale":"2024-03-30T13:10:32.831842+0000","last_undegraded":"2024-03-30T13:10:32.831842+0000","last_fullsized":"2024-03-30T13:10:32.831842+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:50.334220+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:09:50.334220+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:14.996022+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":280,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-03-30T13:10:28.271839+0000","last_change":"2024-03-30T13:09:34.558439+0000","last_active":"2024-03-30T13:10:28.271839+0000","last_peered":"2024-03-30T13:10:28.271839+0000","last_clean":"2024-03-30T13:10:28.271839+0000","last_became_active":"2024-03-30T13:09:27.162655+0000","last_became_peered":"2024-03-30T13:09:27.162655+0000","last_unstale":"2024-03-30T13:10:28.271839+0000","last_undegraded":"2024-03-30T13:10:28.271839+0000","last_fullsized":"2024-03-30T13:10:28.271839+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:09:34.558164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:09:34.558164+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:45.791448+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":181,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-30T13:10:29.283948+0000","last_change":"2024-03-30T13:10:08.609586+0000","last_active":"2024-03-30T13:10:29.283948+0000","last_peered":"2024-03-30T13:10:29.283948+0000","last_clean":"2024-03-30T13:10:29.283948+0000","last_became_active":"2024-03-30T13:08:39.543465+0000","last_became_peered":"2024-03-30T13:08:39.543465+0000","last_unstale":"2024-03-30T13:10:29.283948+0000","last_undegraded":"2024-03-30T13:10:29.283948+0000","last_fullsized":"2024-03-30T13:10:29.283948+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:10:08.609336+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:10:08.609336+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:17.880548+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":189042595,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834127,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52076,"kb_used_data":19932,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94319764,"statfs":{"total":96636764160,"available":96583438336,"internally_reserved":0,"allocated":20410368,"data_stored":17689988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534755,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57240,"kb_used_data":24652,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94314600,"statfs":{"total":96636764160,"available":96578150400,"internally_reserved":0,"allocated":25243648,"data_stored":22536870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":33350120},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824067,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48968,"kb_used_data":16380,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94322872,"statfs":{"total":96636764160,"available":96586620928,"internally_reserved":0,"allocated":16773120,"data_stored":14052740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509623,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42308,"kb_used_data":11124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329532,"statfs":{"total":96636764160,"available":96593440768,"internally_reserved":0,"allocated":11390976,"data_stored":8684198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":346,"seq":1486058684441,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42540,"kb_used_data":10400,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94329300,"statfs":{"total":96636764160,"available":96593203200,"internally_reserved":0,"allocated":10649600,"data_stored":7942822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6045,"internal_metadata":32893027},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509623,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49496,"kb_used_data":18312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322344,"statfs":{"total":96636764160,"available":96586080256,"internally_reserved":0,"allocated":18751488,"data_stored":16037909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":298,"seq":1279900254254,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52524,"kb_used_data":14880,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94319316,"statfs":{"total":96636764160,"available":96582979584,"internally_reserved":0,"allocated":15237120,"data_stored":12509939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10986,"internal_metadata":38524182},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147333,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49032,"kb_used_data":17336,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94322808,"statfs":{"total":96636764160,"available":96586555392,"internally_reserved":0,"allocated":17752064,"data_stored":15024883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:10:34.005 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:10:34.006 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:10:34.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:34.130+0000 7f92d3eae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:34.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:34.142+0000 7f92d3eae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:34.316 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:10:34.316 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":409,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:10:33.324953+0000","last_up_change":"2024-03-30T13:10:17.141416+0000","last_in_change":"2024-03-30T13:10:17.141416+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":245,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":83,"last_clean_end":336,"up_from":346,"up_thru":372,"down_at":338,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2287832000},{"type":"v1","addr":"172.21.15.67:6818","nonce":2287832000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2287832000},{"type":"v1","addr":"172.21.15.67:6821","nonce":2287832000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2287832000},{"type":"v1","addr":"172.21.15.67:6828","nonce":2287832000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2287832000},{"type":"v1","addr":"172.21.15.67:6825","nonce":2287832000}]},"public_addr":"172.21.15.67:6818/2287832000","cluster_addr":"172.21.15.67:6821/2287832000","heartbeat_back_addr":"172.21.15.67:6828/2287832000","heartbeat_front_addr":"172.21.15.67:6825/2287832000","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":349,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":388,"up_from":393,"up_thru":393,"down_at":389,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6801","nonce":3837194718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6803","nonce":3837194718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6807","nonce":3837194718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6805","nonce":3837194718}]},"public_addr":"172.21.15.67:6801/3837194718","cluster_addr":"172.21.15.67:6803/3837194718","heartbeat_back_addr":"172.21.15.67:6807/3837194718","heartbeat_front_addr":"172.21.15.67:6805/3837194718","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":135,"last_clean_end":243,"up_from":250,"up_thru":393,"down_at":244,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6803","nonce":3986522908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6807","nonce":3986522908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6815","nonce":3986522908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6811","nonce":3986522908}]},"public_addr":"172.21.15.82:6803/3986522908","cluster_addr":"172.21.15.82:6807/3986522908","heartbeat_back_addr":"172.21.15.82:6815/3986522908","heartbeat_front_addr":"172.21.15.82:6811/3986522908","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":274,"last_clean_end":314,"up_from":320,"up_thru":393,"down_at":315,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":925466951},{"type":"v1","addr":"172.21.15.82:6817","nonce":925466951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":925466951},{"type":"v1","addr":"172.21.15.82:6819","nonce":925466951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":925466951},{"type":"v1","addr":"172.21.15.82:6823","nonce":925466951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":925466951},{"type":"v1","addr":"172.21.15.82:6821","nonce":925466951}]},"public_addr":"172.21.15.82:6817/925466951","cluster_addr":"172.21.15.82:6819/925466951","heartbeat_back_addr":"172.21.15.82:6823/925466951","heartbeat_front_addr":"172.21.15.82:6821/925466951","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":227,"last_clean_end":363,"up_from":372,"up_thru":393,"down_at":364,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6801","nonce":189641108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6806","nonce":189641108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6814","nonce":189641108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6810","nonce":189641108}]},"public_addr":"172.21.15.82:6801/189641108","cluster_addr":"172.21.15.82:6806/189641108","heartbeat_back_addr":"172.21.15.82:6814/189641108","heartbeat_front_addr":"172.21.15.82:6810/189641108","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":206,"last_clean_end":292,"up_from":298,"up_thru":393,"down_at":293,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3894469331},{"type":"v1","addr":"172.21.15.82:6825","nonce":3894469331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3894469331},{"type":"v1","addr":"172.21.15.82:6827","nonce":3894469331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3894469331},{"type":"v1","addr":"172.21.15.82:6831","nonce":3894469331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3894469331},{"type":"v1","addr":"172.21.15.82:6829","nonce":3894469331}]},"public_addr":"172.21.15.82:6825/3894469331","cluster_addr":"172.21.15.82:6827/3894469331","heartbeat_back_addr":"172.21.15.82:6831/3894469331","heartbeat_front_addr":"172.21.15.82:6829/3894469331","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:09:16.074169+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:09:53.193647+0000","dead_epoch":338},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:09:13.332707+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:10:16.338119+0000","dead_epoch":389},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:09:29.251348+0000","dead_epoch":244},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:09:27.308480+0000","dead_epoch":315},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:09:25.835127+0000","dead_epoch":364},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:09:31.013353+0000","dead_epoch":293}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:10:34.336 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:10:34.336 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:10:34.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:34.450+0000 7f80e1ec3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:34.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:34.466+0000 7f80e1ec3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:34.636 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:10:34.654 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:10:34.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:34.758+0000 7f38cd122640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:34.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:34.782+0000 7f38cd122640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:34.957 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:10:34.957 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:10:34.974 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:10:34.974 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:10:34.975 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:10:34.975 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:10:34.975 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 4, 7, 5, 1, 6, 3] 2024-03-30T13:10:34.975 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:10:34.975 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:10:35.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:35.082+0000 7fd1f6c92640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:35.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:35.094+0000 7fd1f6c92640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:35.266 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:10:35.267 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:10:35.285 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":853,"stamp":"2024-03-30T13:10:34.041723+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":189042595,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":394184,"kb_used_data":133016,"kb_used_omap":44,"kb_used_meta":261011,"kb_avail":754580536,"statfs":{"total":773094113280,"available":772690468864,"internally_reserved":0,"allocated":136208384,"data_stored":114479349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":46068,"internal_metadata":267275276},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001092"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":904,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-30T13:10:30.492174+0000","last_change":"2024-03-30T13:10:16.176071+0000","last_active":"2024-03-30T13:10:30.492174+0000","last_peered":"2024-03-30T13:10:30.492174+0000","last_clean":"2024-03-30T13:10:30.492174+0000","last_became_active":"2024-03-30T13:09:27.168768+0000","last_became_peered":"2024-03-30T13:09:27.168768+0000","last_unstale":"2024-03-30T13:10:30.492174+0000","last_undegraded":"2024-03-30T13:10:30.492174+0000","last_fullsized":"2024-03-30T13:10:30.492174+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:10:16.175659+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:10:16.175659+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:39.530667+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":879,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-30T13:10:30.295653+0000","last_change":"2024-03-30T13:10:18.178182+0000","last_active":"2024-03-30T13:10:30.295653+0000","last_peered":"2024-03-30T13:10:30.295653+0000","last_clean":"2024-03-30T13:10:30.295653+0000","last_became_active":"2024-03-30T13:10:18.176067+0000","last_became_peered":"2024-03-30T13:10:18.176067+0000","last_unstale":"2024-03-30T13:10:30.295653+0000","last_undegraded":"2024-03-30T13:10:30.295653+0000","last_fullsized":"2024-03-30T13:10:30.295653+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:39.428863+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:09:39.428863+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:05.213207+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":502,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-30T13:10:32.832046+0000","last_change":"2024-03-30T13:09:45.261607+0000","last_active":"2024-03-30T13:10:32.832046+0000","last_peered":"2024-03-30T13:10:32.832046+0000","last_clean":"2024-03-30T13:10:32.832046+0000","last_became_active":"2024-03-30T13:08:39.541896+0000","last_became_peered":"2024-03-30T13:08:39.541896+0000","last_unstale":"2024-03-30T13:10:32.832046+0000","last_undegraded":"2024-03-30T13:10:32.832046+0000","last_fullsized":"2024-03-30T13:10:32.832046+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:09:45.261353+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:09:45.261353+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:55.313728+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":472,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-30T13:10:32.831677+0000","last_change":"2024-03-30T13:09:53.320247+0000","last_active":"2024-03-30T13:10:32.831677+0000","last_peered":"2024-03-30T13:10:32.831677+0000","last_clean":"2024-03-30T13:10:32.831677+0000","last_became_active":"2024-03-30T13:09:27.164268+0000","last_became_peered":"2024-03-30T13:09:27.164268+0000","last_unstale":"2024-03-30T13:10:32.831677+0000","last_undegraded":"2024-03-30T13:10:32.831677+0000","last_fullsized":"2024-03-30T13:10:32.831677+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:09:53.319978+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:02.568868+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":242,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-03-30T13:10:28.272119+0000","last_change":"2024-03-30T13:09:56.639400+0000","last_active":"2024-03-30T13:10:28.272119+0000","last_peered":"2024-03-30T13:10:28.272119+0000","last_clean":"2024-03-30T13:10:28.272119+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:10:28.272119+0000","last_undegraded":"2024-03-30T13:10:28.272119+0000","last_fullsized":"2024-03-30T13:10:28.272119+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:09:56.639030+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:59.738230+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":910,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-03-30T13:10:31.307754+0000","last_change":"2024-03-30T13:10:18.181812+0000","last_active":"2024-03-30T13:10:31.307754+0000","last_peered":"2024-03-30T13:10:31.307754+0000","last_clean":"2024-03-30T13:10:31.307754+0000","last_became_active":"2024-03-30T13:10:18.181149+0000","last_became_peered":"2024-03-30T13:10:18.181149+0000","last_unstale":"2024-03-30T13:10:31.307754+0000","last_undegraded":"2024-03-30T13:10:31.307754+0000","last_fullsized":"2024-03-30T13:10:31.307754+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:10:09.027639+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_clean_scrub_stamp":"2024-03-30T13:10:09.027639+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:09.537539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":481,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-30T13:10:32.831802+0000","last_change":"2024-03-30T13:09:42.175053+0000","last_active":"2024-03-30T13:10:32.831802+0000","last_peered":"2024-03-30T13:10:32.831802+0000","last_clean":"2024-03-30T13:10:32.831802+0000","last_became_active":"2024-03-30T13:06:54.155351+0000","last_became_peered":"2024-03-30T13:06:54.155351+0000","last_unstale":"2024-03-30T13:10:32.831802+0000","last_undegraded":"2024-03-30T13:10:32.831802+0000","last_fullsized":"2024-03-30T13:10:32.831802+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:09:42.174783+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:09:42.174783+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:00.350292+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":206,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-30T13:10:29.283937+0000","last_change":"2024-03-30T13:10:23.571319+0000","last_active":"2024-03-30T13:10:29.283937+0000","last_peered":"2024-03-30T13:10:29.283937+0000","last_clean":"2024-03-30T13:10:29.283937+0000","last_became_active":"2024-03-30T13:09:27.546829+0000","last_became_peered":"2024-03-30T13:09:27.546829+0000","last_unstale":"2024-03-30T13:10:29.283937+0000","last_undegraded":"2024-03-30T13:10:29.283937+0000","last_fullsized":"2024-03-30T13:10:29.283937+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:10:23.570911+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:10:23.570911+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:45.997399+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1011,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-03-30T13:10:31.307649+0000","last_change":"2024-03-30T13:10:18.715230+0000","last_active":"2024-03-30T13:10:31.307649+0000","last_peered":"2024-03-30T13:10:31.307649+0000","last_clean":"2024-03-30T13:10:31.307649+0000","last_became_active":"2024-03-30T13:10:18.714034+0000","last_became_peered":"2024-03-30T13:10:18.714034+0000","last_unstale":"2024-03-30T13:10:31.307649+0000","last_undegraded":"2024-03-30T13:10:31.307649+0000","last_fullsized":"2024-03-30T13:10:31.307649+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:09:34.161806+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:08:29.495729+0000","last_clean_scrub_stamp":"2024-03-30T13:09:34.161806+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:43.489690+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":50756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":781,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-30T13:10:30.295379+0000","last_change":"2024-03-30T13:09:27.170807+0000","last_active":"2024-03-30T13:10:30.295379+0000","last_peered":"2024-03-30T13:10:30.295379+0000","last_clean":"2024-03-30T13:10:30.295379+0000","last_became_active":"2024-03-30T13:09:27.169549+0000","last_became_peered":"2024-03-30T13:09:27.169549+0000","last_unstale":"2024-03-30T13:10:30.295379+0000","last_undegraded":"2024-03-30T13:10:30.295379+0000","last_fullsized":"2024-03-30T13:10:30.295379+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:15.464410+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:09:15.464410+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:39.896795+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":911,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-30T13:10:30.491918+0000","last_change":"2024-03-30T13:09:54.854942+0000","last_active":"2024-03-30T13:10:30.491918+0000","last_peered":"2024-03-30T13:10:30.491918+0000","last_clean":"2024-03-30T13:10:30.491918+0000","last_became_active":"2024-03-30T13:09:27.167070+0000","last_became_peered":"2024-03-30T13:09:27.167070+0000","last_unstale":"2024-03-30T13:10:30.491918+0000","last_undegraded":"2024-03-30T13:10:30.491918+0000","last_fullsized":"2024-03-30T13:10:30.491918+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:09:54.854686+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:09:54.854686+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:19.248421+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":976,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-30T13:10:30.491749+0000","last_change":"2024-03-30T13:10:18.181749+0000","last_active":"2024-03-30T13:10:30.491749+0000","last_peered":"2024-03-30T13:10:30.491749+0000","last_clean":"2024-03-30T13:10:30.491749+0000","last_became_active":"2024-03-30T13:10:18.180670+0000","last_became_peered":"2024-03-30T13:10:18.180670+0000","last_unstale":"2024-03-30T13:10:30.491749+0000","last_undegraded":"2024-03-30T13:10:30.491749+0000","last_fullsized":"2024-03-30T13:10:30.491749+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:09:47.821071+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:11.230137+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":280,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-03-30T13:10:28.271819+0000","last_change":"2024-03-30T13:10:18.185077+0000","last_active":"2024-03-30T13:10:28.271819+0000","last_peered":"2024-03-30T13:10:28.271819+0000","last_clean":"2024-03-30T13:10:28.271819+0000","last_became_active":"2024-03-30T13:10:18.183814+0000","last_became_peered":"2024-03-30T13:10:18.183814+0000","last_unstale":"2024-03-30T13:10:28.271819+0000","last_undegraded":"2024-03-30T13:10:28.271819+0000","last_fullsized":"2024-03-30T13:10:28.271819+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:10:13.731448+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:10:13.731448+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:23.829826+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":435,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-30T13:10:32.831842+0000","last_change":"2024-03-30T13:10:18.175121+0000","last_active":"2024-03-30T13:10:32.831842+0000","last_peered":"2024-03-30T13:10:32.831842+0000","last_clean":"2024-03-30T13:10:32.831842+0000","last_became_active":"2024-03-30T13:10:18.174189+0000","last_became_peered":"2024-03-30T13:10:18.174189+0000","last_unstale":"2024-03-30T13:10:32.831842+0000","last_undegraded":"2024-03-30T13:10:32.831842+0000","last_fullsized":"2024-03-30T13:10:32.831842+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:50.334220+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:09:50.334220+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:14.996022+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":280,"reported_epoch":404,"state":"active+clean","last_fresh":"2024-03-30T13:10:28.271839+0000","last_change":"2024-03-30T13:09:34.558439+0000","last_active":"2024-03-30T13:10:28.271839+0000","last_peered":"2024-03-30T13:10:28.271839+0000","last_clean":"2024-03-30T13:10:28.271839+0000","last_became_active":"2024-03-30T13:09:27.162655+0000","last_became_peered":"2024-03-30T13:09:27.162655+0000","last_unstale":"2024-03-30T13:10:28.271839+0000","last_undegraded":"2024-03-30T13:10:28.271839+0000","last_fullsized":"2024-03-30T13:10:28.271839+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:09:34.558164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:09:34.558164+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:45.791448+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":181,"reported_epoch":405,"state":"active+clean","last_fresh":"2024-03-30T13:10:29.283948+0000","last_change":"2024-03-30T13:10:08.609586+0000","last_active":"2024-03-30T13:10:29.283948+0000","last_peered":"2024-03-30T13:10:29.283948+0000","last_clean":"2024-03-30T13:10:29.283948+0000","last_became_active":"2024-03-30T13:08:39.543465+0000","last_became_peered":"2024-03-30T13:08:39.543465+0000","last_unstale":"2024-03-30T13:10:29.283948+0000","last_undegraded":"2024-03-30T13:10:29.283948+0000","last_fullsized":"2024-03-30T13:10:29.283948+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:10:08.609336+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:10:08.609336+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:17.880548+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":189042595,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834127,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52076,"kb_used_data":19932,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94319764,"statfs":{"total":96636764160,"available":96583438336,"internally_reserved":0,"allocated":20410368,"data_stored":17689988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534755,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57240,"kb_used_data":24652,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94314600,"statfs":{"total":96636764160,"available":96578150400,"internally_reserved":0,"allocated":25243648,"data_stored":22536870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":33350120},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824067,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48968,"kb_used_data":16380,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94322872,"statfs":{"total":96636764160,"available":96586620928,"internally_reserved":0,"allocated":16773120,"data_stored":14052740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509623,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42308,"kb_used_data":11124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329532,"statfs":{"total":96636764160,"available":96593440768,"internally_reserved":0,"allocated":11390976,"data_stored":8684198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":346,"seq":1486058684441,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42540,"kb_used_data":10400,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94329300,"statfs":{"total":96636764160,"available":96593203200,"internally_reserved":0,"allocated":10649600,"data_stored":7942822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6045,"internal_metadata":32893027},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509623,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49496,"kb_used_data":18312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322344,"statfs":{"total":96636764160,"available":96586080256,"internally_reserved":0,"allocated":18751488,"data_stored":16037909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":298,"seq":1279900254254,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52524,"kb_used_data":14880,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94319316,"statfs":{"total":96636764160,"available":96582979584,"internally_reserved":0,"allocated":15237120,"data_stored":12509939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10986,"internal_metadata":38524182},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147333,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49032,"kb_used_data":17336,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94322808,"statfs":{"total":96636764160,"available":96586555392,"internally_reserved":0,"allocated":17752064,"data_stored":15024883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:10:35.286 INFO:tasks.thrashosds.thrasher:Choosing PG 1.0 with acting set [7, 0, 6] 2024-03-30T13:10:35.286 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 2, 4, 7, 5, 1, 6, 3] 2024-03-30T13:10:35.286 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-30T13:10:35.286 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:10:35.360 INFO:tasks.ceph.osd.7:Stopped 2024-03-30T13:10:35.360 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 2, 4, 7, 5, 1, 6, 3] 2024-03-30T13:10:35.360 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 7 2024-03-30T13:10:35.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:35.482+0000 7fa864258640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:35.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:35.498+0000 7fa864258640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:35.674 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:10:35.678+0000 7fbb38f08640 -1 mon.a@0(leader).osd e410 definitely_dead 0 2024-03-30T13:10:36.350 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.7. 2024-03-30T13:10:36.381 INFO:tasks.thrashosds.thrasher:dead_osds=[7], live_osds=[0, 2, 4, 5, 1, 6, 3] 2024-03-30T13:10:36.381 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:10:36.381 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:10:36.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:36.498+0000 7f0295175640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:36.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:10:36.510+0000 7f0295175640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:10:36.680 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:10:36.681 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:10:36.699 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":856,"stamp":"2024-03-30T13:10:36.354198+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":189042595,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":341924,"kb_used_data":118388,"kb_used_omap":34,"kb_used_meta":223389,"kb_avail":660260956,"statfs":{"total":676457349120,"available":676107218944,"internally_reserved":0,"allocated":121229312,"data_stored":102084637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35082,"internal_metadata":228751094},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.313007"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":912,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-30T13:10:35.364551+0000","last_change":"2024-03-30T13:10:16.176071+0000","last_active":"2024-03-30T13:10:35.364551+0000","last_peered":"2024-03-30T13:10:35.364551+0000","last_clean":"2024-03-30T13:10:35.364551+0000","last_became_active":"2024-03-30T13:09:27.168768+0000","last_became_peered":"2024-03-30T13:09:27.168768+0000","last_unstale":"2024-03-30T13:10:35.364551+0000","last_undegraded":"2024-03-30T13:10:35.364551+0000","last_fullsized":"2024-03-30T13:10:35.364551+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:10:16.175659+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:10:16.175659+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:39.530667+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":879,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-30T13:10:30.295653+0000","last_change":"2024-03-30T13:10:18.178182+0000","last_active":"2024-03-30T13:10:30.295653+0000","last_peered":"2024-03-30T13:10:30.295653+0000","last_clean":"2024-03-30T13:10:30.295653+0000","last_became_active":"2024-03-30T13:10:18.176067+0000","last_became_peered":"2024-03-30T13:10:18.176067+0000","last_unstale":"2024-03-30T13:10:30.295653+0000","last_undegraded":"2024-03-30T13:10:30.295653+0000","last_fullsized":"2024-03-30T13:10:30.295653+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:39.428863+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:58:30.264658+0000","last_clean_scrub_stamp":"2024-03-30T13:09:39.428863+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:05.213207+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":502,"reported_epoch":408,"state":"stale+active+clean","last_fresh":"2024-03-30T13:10:32.832046+0000","last_change":"2024-03-30T13:09:45.261607+0000","last_active":"2024-03-30T13:10:32.832046+0000","last_peered":"2024-03-30T13:10:32.832046+0000","last_clean":"2024-03-30T13:10:32.832046+0000","last_became_active":"2024-03-30T13:08:39.541896+0000","last_became_peered":"2024-03-30T13:08:39.541896+0000","last_unstale":"2024-03-30T13:10:35.350973+0000","last_undegraded":"2024-03-30T13:10:32.832046+0000","last_fullsized":"2024-03-30T13:10:32.832046+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:09:45.261353+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:09:45.261353+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:55.313728+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":472,"reported_epoch":408,"state":"stale+active+clean","last_fresh":"2024-03-30T13:10:32.831677+0000","last_change":"2024-03-30T13:09:53.320247+0000","last_active":"2024-03-30T13:10:32.831677+0000","last_peered":"2024-03-30T13:10:32.831677+0000","last_clean":"2024-03-30T13:10:32.831677+0000","last_became_active":"2024-03-30T13:09:27.164268+0000","last_became_peered":"2024-03-30T13:09:27.164268+0000","last_unstale":"2024-03-30T13:10:35.350977+0000","last_undegraded":"2024-03-30T13:10:32.831677+0000","last_fullsized":"2024-03-30T13:10:32.831677+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:09:53.319978+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:02.568868+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":252,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-30T13:10:33.931928+0000","last_change":"2024-03-30T13:09:56.639400+0000","last_active":"2024-03-30T13:10:33.931928+0000","last_peered":"2024-03-30T13:10:33.931928+0000","last_clean":"2024-03-30T13:10:33.931928+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:10:33.931928+0000","last_undegraded":"2024-03-30T13:10:33.931928+0000","last_fullsized":"2024-03-30T13:10:33.931928+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:09:56.639030+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:59.738230+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":910,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-03-30T13:10:31.307754+0000","last_change":"2024-03-30T13:10:18.181812+0000","last_active":"2024-03-30T13:10:31.307754+0000","last_peered":"2024-03-30T13:10:31.307754+0000","last_clean":"2024-03-30T13:10:31.307754+0000","last_became_active":"2024-03-30T13:10:18.181149+0000","last_became_peered":"2024-03-30T13:10:18.181149+0000","last_unstale":"2024-03-30T13:10:31.307754+0000","last_undegraded":"2024-03-30T13:10:31.307754+0000","last_fullsized":"2024-03-30T13:10:31.307754+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:10:09.027639+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_clean_scrub_stamp":"2024-03-30T13:10:09.027639+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:09.537539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":481,"reported_epoch":408,"state":"stale+active+clean","last_fresh":"2024-03-30T13:10:32.831802+0000","last_change":"2024-03-30T13:09:42.175053+0000","last_active":"2024-03-30T13:10:32.831802+0000","last_peered":"2024-03-30T13:10:32.831802+0000","last_clean":"2024-03-30T13:10:32.831802+0000","last_became_active":"2024-03-30T13:06:54.155351+0000","last_became_peered":"2024-03-30T13:06:54.155351+0000","last_unstale":"2024-03-30T13:10:35.350982+0000","last_undegraded":"2024-03-30T13:10:32.831802+0000","last_fullsized":"2024-03-30T13:10:32.831802+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:09:42.174783+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:09:42.174783+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:00.350292+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":214,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-30T13:10:33.336023+0000","last_change":"2024-03-30T13:10:23.571319+0000","last_active":"2024-03-30T13:10:33.336023+0000","last_peered":"2024-03-30T13:10:33.336023+0000","last_clean":"2024-03-30T13:10:33.336023+0000","last_became_active":"2024-03-30T13:09:27.546829+0000","last_became_peered":"2024-03-30T13:09:27.546829+0000","last_unstale":"2024-03-30T13:10:33.336023+0000","last_undegraded":"2024-03-30T13:10:33.336023+0000","last_fullsized":"2024-03-30T13:10:33.336023+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:10:23.570911+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:10:23.570911+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:45.997399+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1011,"reported_epoch":407,"state":"active+clean","last_fresh":"2024-03-30T13:10:31.307649+0000","last_change":"2024-03-30T13:10:18.715230+0000","last_active":"2024-03-30T13:10:31.307649+0000","last_peered":"2024-03-30T13:10:31.307649+0000","last_clean":"2024-03-30T13:10:31.307649+0000","last_became_active":"2024-03-30T13:10:18.714034+0000","last_became_peered":"2024-03-30T13:10:18.714034+0000","last_unstale":"2024-03-30T13:10:31.307649+0000","last_undegraded":"2024-03-30T13:10:31.307649+0000","last_fullsized":"2024-03-30T13:10:31.307649+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:09:34.161806+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:08:29.495729+0000","last_clean_scrub_stamp":"2024-03-30T13:09:34.161806+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:43.489690+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":50756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":781,"reported_epoch":406,"state":"active+clean","last_fresh":"2024-03-30T13:10:30.295379+0000","last_change":"2024-03-30T13:09:27.170807+0000","last_active":"2024-03-30T13:10:30.295379+0000","last_peered":"2024-03-30T13:10:30.295379+0000","last_clean":"2024-03-30T13:10:30.295379+0000","last_became_active":"2024-03-30T13:09:27.169549+0000","last_became_peered":"2024-03-30T13:09:27.169549+0000","last_unstale":"2024-03-30T13:10:30.295379+0000","last_undegraded":"2024-03-30T13:10:30.295379+0000","last_fullsized":"2024-03-30T13:10:30.295379+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:15.464410+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:09:15.464410+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:39.896795+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":919,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-30T13:10:35.364398+0000","last_change":"2024-03-30T13:09:54.854942+0000","last_active":"2024-03-30T13:10:35.364398+0000","last_peered":"2024-03-30T13:10:35.364398+0000","last_clean":"2024-03-30T13:10:35.364398+0000","last_became_active":"2024-03-30T13:09:27.167070+0000","last_became_peered":"2024-03-30T13:09:27.167070+0000","last_unstale":"2024-03-30T13:10:35.364398+0000","last_undegraded":"2024-03-30T13:10:35.364398+0000","last_fullsized":"2024-03-30T13:10:35.364398+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:09:54.854686+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:09:54.854686+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:19.248421+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":984,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-30T13:10:35.364384+0000","last_change":"2024-03-30T13:10:18.181749+0000","last_active":"2024-03-30T13:10:35.364384+0000","last_peered":"2024-03-30T13:10:35.364384+0000","last_clean":"2024-03-30T13:10:35.364384+0000","last_became_active":"2024-03-30T13:10:18.180670+0000","last_became_peered":"2024-03-30T13:10:18.180670+0000","last_unstale":"2024-03-30T13:10:35.364384+0000","last_undegraded":"2024-03-30T13:10:35.364384+0000","last_fullsized":"2024-03-30T13:10:35.364384+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:09:47.821071+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:11.230137+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":290,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-30T13:10:33.931704+0000","last_change":"2024-03-30T13:10:18.185077+0000","last_active":"2024-03-30T13:10:33.931704+0000","last_peered":"2024-03-30T13:10:33.931704+0000","last_clean":"2024-03-30T13:10:33.931704+0000","last_became_active":"2024-03-30T13:10:18.183814+0000","last_became_peered":"2024-03-30T13:10:18.183814+0000","last_unstale":"2024-03-30T13:10:33.931704+0000","last_undegraded":"2024-03-30T13:10:33.931704+0000","last_fullsized":"2024-03-30T13:10:33.931704+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:10:13.731448+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:10:13.731448+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:23.829826+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":435,"reported_epoch":408,"state":"stale+active+clean","last_fresh":"2024-03-30T13:10:32.831842+0000","last_change":"2024-03-30T13:10:18.175121+0000","last_active":"2024-03-30T13:10:32.831842+0000","last_peered":"2024-03-30T13:10:32.831842+0000","last_clean":"2024-03-30T13:10:32.831842+0000","last_became_active":"2024-03-30T13:10:18.174189+0000","last_became_peered":"2024-03-30T13:10:18.174189+0000","last_unstale":"2024-03-30T13:10:35.350988+0000","last_undegraded":"2024-03-30T13:10:32.831842+0000","last_fullsized":"2024-03-30T13:10:32.831842+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:50.334220+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:09:50.334220+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:14.996022+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":290,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-30T13:10:33.932116+0000","last_change":"2024-03-30T13:09:34.558439+0000","last_active":"2024-03-30T13:10:33.932116+0000","last_peered":"2024-03-30T13:10:33.932116+0000","last_clean":"2024-03-30T13:10:33.932116+0000","last_became_active":"2024-03-30T13:09:27.162655+0000","last_became_peered":"2024-03-30T13:09:27.162655+0000","last_unstale":"2024-03-30T13:10:33.932116+0000","last_undegraded":"2024-03-30T13:10:33.932116+0000","last_fullsized":"2024-03-30T13:10:33.932116+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:09:34.558164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-30T12:55:42.900074+0000","last_clean_scrub_stamp":"2024-03-30T13:09:34.558164+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:45.791448+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":189,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-30T13:10:33.336038+0000","last_change":"2024-03-30T13:10:08.609586+0000","last_active":"2024-03-30T13:10:33.336038+0000","last_peered":"2024-03-30T13:10:33.336038+0000","last_clean":"2024-03-30T13:10:33.336038+0000","last_became_active":"2024-03-30T13:08:39.543465+0000","last_became_peered":"2024-03-30T13:08:39.543465+0000","last_unstale":"2024-03-30T13:10:33.336038+0000","last_undegraded":"2024-03-30T13:10:33.336038+0000","last_fullsized":"2024-03-30T13:10:33.336038+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:10:08.609336+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:10:08.609336+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:17.880548+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":189042595,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834128,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52116,"kb_used_data":19968,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94319724,"statfs":{"total":96636764160,"available":96583397376,"internally_reserved":0,"allocated":20447232,"data_stored":17706449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534756,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57292,"kb_used_data":24700,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94314548,"statfs":{"total":96636764160,"available":96578097152,"internally_reserved":0,"allocated":25292800,"data_stored":22558818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":33350120},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824067,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48968,"kb_used_data":16380,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94322872,"statfs":{"total":96636764160,"available":96586620928,"internally_reserved":0,"allocated":16773120,"data_stored":14052740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":33350129},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509624,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42368,"kb_used_data":11184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329472,"statfs":{"total":96636764160,"available":96593379328,"internally_reserved":0,"allocated":11452416,"data_stored":8711633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":346,"seq":1486058684442,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42608,"kb_used_data":10460,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94329232,"statfs":{"total":96636764160,"available":96593133568,"internally_reserved":0,"allocated":10711040,"data_stored":7970257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6045,"internal_metadata":32893027},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509624,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49540,"kb_used_data":18360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322300,"statfs":{"total":96636764160,"available":96586035200,"internally_reserved":0,"allocated":18800640,"data_stored":16059857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147333,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49032,"kb_used_data":17336,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94322808,"statfs":{"total":96636764160,"available":96586555392,"internally_reserved":0,"allocated":17752064,"data_stored":15024883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:10:36.700 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:11:01.701 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:11:01.702 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:11:01.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:01.833+0000 7f5886985640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:01.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:01.845+0000 7f5886985640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:02.018 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:11:02.019 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:11:02.037 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":874,"stamp":"2024-03-30T13:11:00.049131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":204588872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":356772,"kb_used_data":129104,"kb_used_omap":34,"kb_used_meta":227549,"kb_avail":660246108,"statfs":{"total":676457349120,"available":676092014592,"internally_reserved":0,"allocated":132202496,"data_stored":112680507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35082,"internal_metadata":233010934},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002225"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":924,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-30T13:10:53.234341+0000","last_change":"2024-03-30T13:10:16.176071+0000","last_active":"2024-03-30T13:10:53.234341+0000","last_peered":"2024-03-30T13:10:53.234341+0000","last_clean":"2024-03-30T13:10:53.234341+0000","last_became_active":"2024-03-30T13:09:27.168768+0000","last_became_peered":"2024-03-30T13:09:27.168768+0000","last_unstale":"2024-03-30T13:10:53.234341+0000","last_undegraded":"2024-03-30T13:10:53.234341+0000","last_fullsized":"2024-03-30T13:10:53.234341+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:10:16.175659+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:10:16.175659+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:39.530667+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":916,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-30T13:10:53.235009+0000","last_change":"2024-03-30T13:10:44.578324+0000","last_active":"2024-03-30T13:10:53.235009+0000","last_peered":"2024-03-30T13:10:53.235009+0000","last_clean":"2024-03-30T13:10:53.235009+0000","last_became_active":"2024-03-30T13:10:37.369663+0000","last_became_peered":"2024-03-30T13:10:37.369663+0000","last_unstale":"2024-03-30T13:10:53.235009+0000","last_undegraded":"2024-03-30T13:10:53.235009+0000","last_fullsized":"2024-03-30T13:10:53.235009+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_clean_scrub_stamp":"2024-03-30T13:10:44.578038+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:57.200866+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10621892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":37,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-30T13:10:59.147648+0000","last_change":"2024-03-30T13:10:54.353572+0000","last_active":"2024-03-30T13:10:59.147648+0000","last_peered":"2024-03-30T13:10:59.147648+0000","last_clean":"2024-03-30T13:10:59.147648+0000","last_became_active":"2024-03-30T13:10:37.371436+0000","last_became_peered":"2024-03-30T13:10:37.371436+0000","last_unstale":"2024-03-30T13:10:59.147648+0000","last_undegraded":"2024-03-30T13:10:59.147648+0000","last_fullsized":"2024-03-30T13:10:59.147648+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:10:54.353314+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:10:54.353314+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:10.603010+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":27,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-30T13:10:59.146017+0000","last_change":"2024-03-30T13:10:37.420273+0000","last_active":"2024-03-30T13:10:59.146017+0000","last_peered":"2024-03-30T13:10:59.146017+0000","last_clean":"2024-03-30T13:10:59.146017+0000","last_became_active":"2024-03-30T13:10:37.368351+0000","last_became_peered":"2024-03-30T13:10:37.368351+0000","last_unstale":"2024-03-30T13:10:59.146017+0000","last_undegraded":"2024-03-30T13:10:59.146017+0000","last_fullsized":"2024-03-30T13:10:59.146017+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:09:53.319978+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:07.872517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":266,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-30T13:10:53.134646+0000","last_change":"2024-03-30T13:09:56.639400+0000","last_active":"2024-03-30T13:10:53.134646+0000","last_peered":"2024-03-30T13:10:53.134646+0000","last_clean":"2024-03-30T13:10:53.134646+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:10:53.134646+0000","last_undegraded":"2024-03-30T13:10:53.134646+0000","last_fullsized":"2024-03-30T13:10:53.134646+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:09:56.639030+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:10:59.738230+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":928,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-30T13:10:53.235852+0000","last_change":"2024-03-30T13:10:18.181812+0000","last_active":"2024-03-30T13:10:53.235852+0000","last_peered":"2024-03-30T13:10:53.235852+0000","last_clean":"2024-03-30T13:10:53.235852+0000","last_became_active":"2024-03-30T13:10:18.181149+0000","last_became_peered":"2024-03-30T13:10:18.181149+0000","last_unstale":"2024-03-30T13:10:53.235852+0000","last_undegraded":"2024-03-30T13:10:53.235852+0000","last_fullsized":"2024-03-30T13:10:53.235852+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:10:09.027639+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_clean_scrub_stamp":"2024-03-30T13:10:09.027639+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:09.537539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":40,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-30T13:10:53.234863+0000","last_change":"2024-03-30T13:10:37.506285+0000","last_active":"2024-03-30T13:10:53.234863+0000","last_peered":"2024-03-30T13:10:53.234863+0000","last_clean":"2024-03-30T13:10:53.234863+0000","last_became_active":"2024-03-30T13:10:37.369895+0000","last_became_peered":"2024-03-30T13:10:37.369895+0000","last_unstale":"2024-03-30T13:10:53.234863+0000","last_undegraded":"2024-03-30T13:10:53.234863+0000","last_fullsized":"2024-03-30T13:10:53.234863+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:09:42.174783+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:09:42.174783+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:01.148661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":231,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-30T13:10:59.146023+0000","last_change":"2024-03-30T13:10:23.571319+0000","last_active":"2024-03-30T13:10:59.146023+0000","last_peered":"2024-03-30T13:10:59.146023+0000","last_clean":"2024-03-30T13:10:59.146023+0000","last_became_active":"2024-03-30T13:09:27.546829+0000","last_became_peered":"2024-03-30T13:09:27.546829+0000","last_unstale":"2024-03-30T13:10:59.146023+0000","last_undegraded":"2024-03-30T13:10:59.146023+0000","last_fullsized":"2024-03-30T13:10:59.146023+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:10:23.570911+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:10:23.570911+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:45.997399+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1048,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-30T13:10:53.235674+0000","last_change":"2024-03-30T13:10:52.360997+0000","last_active":"2024-03-30T13:10:53.235674+0000","last_peered":"2024-03-30T13:10:53.235674+0000","last_clean":"2024-03-30T13:10:53.235674+0000","last_became_active":"2024-03-30T13:10:37.375848+0000","last_became_peered":"2024-03-30T13:10:37.375848+0000","last_unstale":"2024-03-30T13:10:53.235674+0000","last_undegraded":"2024-03-30T13:10:53.235674+0000","last_fullsized":"2024-03-30T13:10:53.235674+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:10:52.360707+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:11.018467+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":55832128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":809,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-30T13:10:53.234905+0000","last_change":"2024-03-30T13:10:41.530383+0000","last_active":"2024-03-30T13:10:53.234905+0000","last_peered":"2024-03-30T13:10:53.234905+0000","last_clean":"2024-03-30T13:10:53.234905+0000","last_became_active":"2024-03-30T13:09:27.169549+0000","last_became_peered":"2024-03-30T13:09:27.169549+0000","last_unstale":"2024-03-30T13:10:53.234905+0000","last_undegraded":"2024-03-30T13:10:53.234905+0000","last_fullsized":"2024-03-30T13:10:53.234905+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:10:41.529990+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:10:41.529990+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:04.044498+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":931,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-30T13:10:53.234299+0000","last_change":"2024-03-30T13:09:54.854942+0000","last_active":"2024-03-30T13:10:53.234299+0000","last_peered":"2024-03-30T13:10:53.234299+0000","last_clean":"2024-03-30T13:10:53.234299+0000","last_became_active":"2024-03-30T13:09:27.167070+0000","last_became_peered":"2024-03-30T13:09:27.167070+0000","last_unstale":"2024-03-30T13:10:53.234299+0000","last_undegraded":"2024-03-30T13:10:53.234299+0000","last_fullsized":"2024-03-30T13:10:53.234299+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:09:54.854686+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:09:54.854686+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:19.248421+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":996,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-30T13:10:53.234288+0000","last_change":"2024-03-30T13:10:18.181749+0000","last_active":"2024-03-30T13:10:53.234288+0000","last_peered":"2024-03-30T13:10:53.234288+0000","last_clean":"2024-03-30T13:10:53.234288+0000","last_became_active":"2024-03-30T13:10:18.180670+0000","last_became_peered":"2024-03-30T13:10:18.180670+0000","last_unstale":"2024-03-30T13:10:53.234288+0000","last_undegraded":"2024-03-30T13:10:53.234288+0000","last_fullsized":"2024-03-30T13:10:53.234288+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:09:47.821071+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:11.230137+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":304,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-30T13:10:53.134254+0000","last_change":"2024-03-30T13:10:18.185077+0000","last_active":"2024-03-30T13:10:53.134254+0000","last_peered":"2024-03-30T13:10:53.134254+0000","last_clean":"2024-03-30T13:10:53.134254+0000","last_became_active":"2024-03-30T13:10:18.183814+0000","last_became_peered":"2024-03-30T13:10:18.183814+0000","last_unstale":"2024-03-30T13:10:53.134254+0000","last_undegraded":"2024-03-30T13:10:53.134254+0000","last_fullsized":"2024-03-30T13:10:53.134254+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:10:13.731448+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:10:13.731448+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:23.829826+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":23,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-30T13:10:53.134289+0000","last_change":"2024-03-30T13:10:37.407532+0000","last_active":"2024-03-30T13:10:53.134289+0000","last_peered":"2024-03-30T13:10:53.134289+0000","last_clean":"2024-03-30T13:10:53.134289+0000","last_became_active":"2024-03-30T13:10:37.373818+0000","last_became_peered":"2024-03-30T13:10:37.373818+0000","last_unstale":"2024-03-30T13:10:53.134289+0000","last_undegraded":"2024-03-30T13:10:53.134289+0000","last_fullsized":"2024-03-30T13:10:53.134289+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:09:50.334220+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:09:50.334220+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:15.864648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":312,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-03-30T13:10:53.134295+0000","last_change":"2024-03-30T13:10:48.606161+0000","last_active":"2024-03-30T13:10:53.134295+0000","last_peered":"2024-03-30T13:10:53.134295+0000","last_clean":"2024-03-30T13:10:53.134295+0000","last_became_active":"2024-03-30T13:09:27.162655+0000","last_became_peered":"2024-03-30T13:09:27.162655+0000","last_unstale":"2024-03-30T13:10:53.134295+0000","last_undegraded":"2024-03-30T13:10:53.134295+0000","last_fullsized":"2024-03-30T13:10:53.134295+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:10:48.605866+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:55.059047+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":206,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-03-30T13:10:59.146152+0000","last_change":"2024-03-30T13:10:08.609586+0000","last_active":"2024-03-30T13:10:59.146152+0000","last_peered":"2024-03-30T13:10:59.146152+0000","last_clean":"2024-03-30T13:10:59.146152+0000","last_became_active":"2024-03-30T13:08:39.543465+0000","last_became_peered":"2024-03-30T13:08:39.543465+0000","last_unstale":"2024-03-30T13:10:59.146152+0000","last_undegraded":"2024-03-30T13:10:59.146152+0000","last_fullsized":"2024-03-30T13:10:59.146152+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:10:08.609336+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:10:08.609336+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:17.880548+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":204588872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":107036672,"data_stored":107036672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834132,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52192,"kb_used_data":20052,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94319648,"statfs":{"total":96636764160,"available":96583319552,"internally_reserved":0,"allocated":20533248,"data_stored":17745358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534761,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58024,"kb_used_data":25436,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94313816,"statfs":{"total":96636764160,"available":96577347584,"internally_reserved":0,"allocated":26046464,"data_stored":23258574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":33350120},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824072,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53248,"kb_used_data":16500,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94318592,"statfs":{"total":96636764160,"available":96582238208,"internally_reserved":0,"allocated":16896000,"data_stored":14108110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":37609969},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509629,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48556,"kb_used_data":17376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323284,"statfs":{"total":96636764160,"available":96587042816,"internally_reserved":0,"allocated":17793024,"data_stored":14998375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":346,"seq":1486058684447,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45992,"kb_used_data":13852,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94325848,"statfs":{"total":96636764160,"available":96589668352,"internally_reserved":0,"allocated":14184448,"data_stored":11396558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6045,"internal_metadata":32893027},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509629,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49624,"kb_used_data":18444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322216,"statfs":{"total":96636764160,"available":96585949184,"internally_reserved":0,"allocated":18886656,"data_stored":16098766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147338,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49136,"kb_used_data":17444,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94322704,"statfs":{"total":96636764160,"available":96586448896,"internally_reserved":0,"allocated":17862656,"data_stored":15074766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9121792,"data_stored":9121792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12718080,"data_stored":12718080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20983808,"data_stored":20983808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:11:02.038 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:11:02.038 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:11:02.038 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2024-03-30T13:11:02.038 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-30T13:11:02.039 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-30T13:11:02.041 INFO:tasks.ceph.osd.7:Started 2024-03-30T13:11:02.042 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-30T13:11:02.110 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:11:02.108+0000 7f4470f7c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:02.117 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:11:02.116+0000 7f4470f7c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:02.117 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:11:02.116+0000 7f4470f7c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:02.166 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T13:11:02.166 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T13:11:02.166 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T13:11:02.166 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T13:11:02.166 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T13:11:02.166 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T13:11:02.166 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-03-30T13:11:02.167 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-03-30T13:11:02.167 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-03-30T13:11:02.167 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-03-30T13:11:02.167 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-03-30T13:11:02.167 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T13:11:02.168 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:11:02.169 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-03-30T13:11:02.650 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:11:02.648+0000 7f4470f7c7c0 -1 Falling back to public interface 2024-03-30T13:11:06.433 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:11:06.432+0000 7f4470f7c7c0 -1 osd.7 409 log_to_monitors true 2024-03-30T13:11:07.169 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-30T13:11:07.237 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:11:07.236+0000 7f4464d49640 -1 osd.7 409 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:11:07.301 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T13:11:07.302 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T13:11:07.302 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T13:11:07.302 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T13:11:07.315 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:11:07.315 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-03-30T13:11:07.315 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 7 2024-03-30T13:11:07.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:07.445+0000 7f9972efa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:07.459 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:11:07.456+0000 7f445f990640 -1 osd.7 409 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:11:07.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:07.461+0000 7f9972efa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:07.643 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:11:07.645+0000 7fbb38f08640 -1 mon.a@0(leader).osd e418 definitely_dead 0 2024-03-30T13:11:08.188 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.7. 2024-03-30T13:11:08.217 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-03-30T13:11:08.217 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:11:08.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:08.333+0000 7fbf42b66640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:08.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:08.349+0000 7fbf42b66640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:11:09.232 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:11:09.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:09.349+0000 7f0756adc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:09.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:09.365+0000 7f0756adc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:10.218 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:11:10.240 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:11:10.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:10.353+0000 7f01d781d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:10.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:10.373+0000 7f01d781d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:11.221 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:11:11.245 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:11:11.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:11.365+0000 7fe9f46cd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:11.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:11.381+0000 7fe9f46cd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:12.223 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:11:12.252 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:11:12.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:12.377+0000 7f5dc7154640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:12.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:12.393+0000 7f5dc7154640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:13.240 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:11:13.265 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:11:13.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:13.385+0000 7fc8cc438640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:13.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:13.401+0000 7fc8cc438640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:14.236 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:11:14.261 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:11:14.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:14.380+0000 7f1fbfa5a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:14.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:14.396+0000 7f1fbfa5a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:15.256 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:11:15.284 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:11:15.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:15.404+0000 7f018684f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:15.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:15.420+0000 7f018684f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:16.271 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:11:16.296 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:11:16.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:16.416+0000 7fcdba4c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:16.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:16.432+0000 7fcdba4c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:17.284 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:11:17.314 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:11:17.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:17.432+0000 7f5f05545640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:17.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:17.448+0000 7f5f05545640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:18.306 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:11:18.330 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:11:18.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:18.448+0000 7fbd3ae47640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:18.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:18.464+0000 7fbd3ae47640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:19.319 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:11:19.345 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:11:19.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:19.464+0000 7f65cc9ed640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:19.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:19.480+0000 7f65cc9ed640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:20.342 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:11:20.367 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:11:20.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:20.484+0000 7ff95d4e8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:20.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:20.500+0000 7ff95d4e8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:21.337 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:11:21.365 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:11:21.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:21.484+0000 7f35868fb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:21.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:21.500+0000 7f35868fb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:22.356 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:11:22.379 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:11:22.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:22.496+0000 7fc1b7f6d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:22.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:22.516+0000 7fc1b7f6d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:23.358 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:11:23.385 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:11:23.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:23.508+0000 7fc9e0e60640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:23.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:23.524+0000 7fc9e0e60640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:24.373 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:11:24.397 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:11:24.397 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:11:24.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:24.520+0000 7fd19c2ea640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:24.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:24.536+0000 7fd19c2ea640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:24.711 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:11:24.712 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:11:24.728 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":904,"stamp":"2024-03-30T13:11:24.371477+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":189698068,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":400808,"kb_used_data":135452,"kb_used_omap":46,"kb_used_meta":265169,"kb_avail":754573912,"statfs":{"total":773094113280,"available":772683685888,"internally_reserved":0,"allocated":138702848,"data_stored":115610018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47716,"internal_metadata":271533468},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.029565"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":953,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-03-30T13:11:20.349576+0000","last_change":"2024-03-30T13:10:16.176071+0000","last_active":"2024-03-30T13:11:20.349576+0000","last_peered":"2024-03-30T13:11:20.349576+0000","last_clean":"2024-03-30T13:11:20.349576+0000","last_became_active":"2024-03-30T13:09:27.168768+0000","last_became_peered":"2024-03-30T13:09:27.168768+0000","last_unstale":"2024-03-30T13:11:20.349576+0000","last_undegraded":"2024-03-30T13:11:20.349576+0000","last_fullsized":"2024-03-30T13:11:20.349576+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:10:16.175659+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:10:16.175659+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:39.530667+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":954,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-30T13:11:21.345954+0000","last_change":"2024-03-30T13:11:09.207574+0000","last_active":"2024-03-30T13:11:21.345954+0000","last_peered":"2024-03-30T13:11:21.345954+0000","last_clean":"2024-03-30T13:11:21.345954+0000","last_became_active":"2024-03-30T13:11:09.206259+0000","last_became_peered":"2024-03-30T13:11:09.206259+0000","last_unstale":"2024-03-30T13:11:21.345954+0000","last_undegraded":"2024-03-30T13:11:21.345954+0000","last_fullsized":"2024-03-30T13:11:21.345954+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_clean_scrub_stamp":"2024-03-30T13:10:44.578038+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:10.667889+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10621892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":500,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-30T13:11:22.362548+0000","last_change":"2024-03-30T13:11:09.210942+0000","last_active":"2024-03-30T13:11:22.362548+0000","last_peered":"2024-03-30T13:11:22.362548+0000","last_clean":"2024-03-30T13:11:22.362548+0000","last_became_active":"2024-03-30T13:11:09.209738+0000","last_became_peered":"2024-03-30T13:11:09.209738+0000","last_unstale":"2024-03-30T13:11:22.362548+0000","last_undegraded":"2024-03-30T13:11:22.362548+0000","last_fullsized":"2024-03-30T13:11:22.362548+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:10:54.353314+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:10:54.353314+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:58.747219+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":482,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-30T13:11:22.362167+0000","last_change":"2024-03-30T13:11:10.457971+0000","last_active":"2024-03-30T13:11:22.362167+0000","last_peered":"2024-03-30T13:11:22.362167+0000","last_clean":"2024-03-30T13:11:22.362167+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:11:22.362167+0000","last_undegraded":"2024-03-30T13:11:22.362167+0000","last_fullsized":"2024-03-30T13:11:22.362167+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:10.457629+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:11:10.457629+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:22.474745+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":303,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-30T13:11:21.405707+0000","last_change":"2024-03-30T13:11:02.527083+0000","last_active":"2024-03-30T13:11:21.405707+0000","last_peered":"2024-03-30T13:11:21.405707+0000","last_clean":"2024-03-30T13:11:21.405707+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:11:21.405707+0000","last_undegraded":"2024-03-30T13:11:21.405707+0000","last_fullsized":"2024-03-30T13:11:21.405707+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:11:02.526699+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:11:02.526699+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:19.924188+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":965,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-30T13:11:21.345506+0000","last_change":"2024-03-30T13:11:12.454138+0000","last_active":"2024-03-30T13:11:21.345506+0000","last_peered":"2024-03-30T13:11:21.345506+0000","last_clean":"2024-03-30T13:11:21.345506+0000","last_became_active":"2024-03-30T13:10:18.181149+0000","last_became_peered":"2024-03-30T13:10:18.181149+0000","last_unstale":"2024-03-30T13:11:21.345506+0000","last_undegraded":"2024-03-30T13:11:21.345506+0000","last_fullsized":"2024-03-30T13:11:21.345506+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:11:12.454045+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_clean_scrub_stamp":"2024-03-30T13:11:12.454045+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:14.977683+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":491,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-30T13:11:22.362199+0000","last_change":"2024-03-30T13:11:09.314561+0000","last_active":"2024-03-30T13:11:22.362199+0000","last_peered":"2024-03-30T13:11:22.362199+0000","last_clean":"2024-03-30T13:11:22.362199+0000","last_became_active":"2024-03-30T13:11:09.214741+0000","last_became_peered":"2024-03-30T13:11:09.214741+0000","last_unstale":"2024-03-30T13:11:22.362199+0000","last_undegraded":"2024-03-30T13:11:22.362199+0000","last_fullsized":"2024-03-30T13:11:22.362199+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:11:03.485902+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:11:03.485902+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:27.208222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":257,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-30T13:11:19.320177+0000","last_change":"2024-03-30T13:10:23.571319+0000","last_active":"2024-03-30T13:11:19.320177+0000","last_peered":"2024-03-30T13:11:19.320177+0000","last_clean":"2024-03-30T13:11:19.320177+0000","last_became_active":"2024-03-30T13:09:27.546829+0000","last_became_peered":"2024-03-30T13:09:27.546829+0000","last_unstale":"2024-03-30T13:11:19.320177+0000","last_undegraded":"2024-03-30T13:11:19.320177+0000","last_fullsized":"2024-03-30T13:11:19.320177+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:10:23.570911+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:10:23.570911+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:45.997399+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1083,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-30T13:11:21.345341+0000","last_change":"2024-03-30T13:11:09.213327+0000","last_active":"2024-03-30T13:11:21.345341+0000","last_peered":"2024-03-30T13:11:21.345341+0000","last_clean":"2024-03-30T13:11:21.345341+0000","last_became_active":"2024-03-30T13:11:09.212712+0000","last_became_peered":"2024-03-30T13:11:09.212712+0000","last_unstale":"2024-03-30T13:11:21.345341+0000","last_undegraded":"2024-03-30T13:11:21.345341+0000","last_fullsized":"2024-03-30T13:11:21.345341+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:10:52.360707+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:54.603559+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":55832128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":842,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-30T13:11:21.345943+0000","last_change":"2024-03-30T13:10:41.530383+0000","last_active":"2024-03-30T13:11:21.345943+0000","last_peered":"2024-03-30T13:11:21.345943+0000","last_clean":"2024-03-30T13:11:21.345943+0000","last_became_active":"2024-03-30T13:09:27.169549+0000","last_became_peered":"2024-03-30T13:09:27.169549+0000","last_unstale":"2024-03-30T13:11:21.345943+0000","last_undegraded":"2024-03-30T13:11:21.345943+0000","last_fullsized":"2024-03-30T13:11:21.345943+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:10:41.529990+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:10:41.529990+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:04.044498+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":968,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-03-30T13:11:20.349321+0000","last_change":"2024-03-30T13:11:20.346373+0000","last_active":"2024-03-30T13:11:20.349321+0000","last_peered":"2024-03-30T13:11:20.349321+0000","last_clean":"2024-03-30T13:11:20.349321+0000","last_became_active":"2024-03-30T13:09:27.167070+0000","last_became_peered":"2024-03-30T13:09:27.167070+0000","last_unstale":"2024-03-30T13:11:20.349321+0000","last_undegraded":"2024-03-30T13:11:20.349321+0000","last_fullsized":"2024-03-30T13:11:20.349321+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:11:20.345991+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:11:20.345991+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:24.270128+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1033,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-03-30T13:11:20.349343+0000","last_change":"2024-03-30T13:11:13.269830+0000","last_active":"2024-03-30T13:11:20.349343+0000","last_peered":"2024-03-30T13:11:20.349343+0000","last_clean":"2024-03-30T13:11:20.349343+0000","last_became_active":"2024-03-30T13:10:18.180670+0000","last_became_peered":"2024-03-30T13:10:18.180670+0000","last_unstale":"2024-03-30T13:11:20.349343+0000","last_undegraded":"2024-03-30T13:11:20.349343+0000","last_fullsized":"2024-03-30T13:11:20.349343+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:11:13.269449+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:11:13.269449+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:32.598288+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":333,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-30T13:11:21.405683+0000","last_change":"2024-03-30T13:10:18.185077+0000","last_active":"2024-03-30T13:11:21.405683+0000","last_peered":"2024-03-30T13:11:21.405683+0000","last_clean":"2024-03-30T13:11:21.405683+0000","last_became_active":"2024-03-30T13:10:18.183814+0000","last_became_peered":"2024-03-30T13:10:18.183814+0000","last_unstale":"2024-03-30T13:11:21.405683+0000","last_undegraded":"2024-03-30T13:11:21.405683+0000","last_fullsized":"2024-03-30T13:11:21.405683+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:10:13.731448+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:10:13.731448+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:23.829826+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":453,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-30T13:11:22.362191+0000","last_change":"2024-03-30T13:11:15.422620+0000","last_active":"2024-03-30T13:11:22.362191+0000","last_peered":"2024-03-30T13:11:22.362191+0000","last_clean":"2024-03-30T13:11:22.362191+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:11:22.362191+0000","last_undegraded":"2024-03-30T13:11:22.362191+0000","last_fullsized":"2024-03-30T13:11:22.362191+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:11:15.422350+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:11:15.422350+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:17.684567+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":341,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-30T13:11:21.405724+0000","last_change":"2024-03-30T13:10:48.606161+0000","last_active":"2024-03-30T13:11:21.405724+0000","last_peered":"2024-03-30T13:11:21.405724+0000","last_clean":"2024-03-30T13:11:21.405724+0000","last_became_active":"2024-03-30T13:09:27.162655+0000","last_became_peered":"2024-03-30T13:09:27.162655+0000","last_unstale":"2024-03-30T13:11:21.405724+0000","last_undegraded":"2024-03-30T13:11:21.405724+0000","last_fullsized":"2024-03-30T13:11:21.405724+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:10:48.605866+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:55.059047+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":232,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-30T13:11:19.320192+0000","last_change":"2024-03-30T13:10:08.609586+0000","last_active":"2024-03-30T13:11:19.320192+0000","last_peered":"2024-03-30T13:11:19.320192+0000","last_clean":"2024-03-30T13:11:19.320192+0000","last_became_active":"2024-03-30T13:08:39.543465+0000","last_became_peered":"2024-03-30T13:08:39.543465+0000","last_unstale":"2024-03-30T13:11:19.320192+0000","last_undegraded":"2024-03-30T13:11:19.320192+0000","last_fullsized":"2024-03-30T13:11:19.320192+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:10:08.609336+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:10:08.609336+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":189698068,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834137,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52360,"kb_used_data":20220,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94319480,"statfs":{"total":96636764160,"available":96583147520,"internally_reserved":0,"allocated":20705280,"data_stored":17823777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534766,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57588,"kb_used_data":24988,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94314252,"statfs":{"total":96636764160,"available":96577794048,"internally_reserved":0,"allocated":25587712,"data_stored":22692607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":33350120},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824077,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53436,"kb_used_data":16680,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94318404,"statfs":{"total":96636764160,"available":96582045696,"internally_reserved":0,"allocated":17080320,"data_stored":14192016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":37609969},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509633,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42604,"kb_used_data":11424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329236,"statfs":{"total":96636764160,"available":96593137664,"internally_reserved":0,"allocated":11698176,"data_stored":8823474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":346,"seq":1486058684451,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42848,"kb_used_data":10700,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94328992,"statfs":{"total":96636764160,"available":96592887808,"internally_reserved":0,"allocated":10956800,"data_stored":8082098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6045,"internal_metadata":32893027},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509633,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49796,"kb_used_data":18612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322044,"statfs":{"total":96636764160,"available":96585773056,"internally_reserved":0,"allocated":19058688,"data_stored":16177185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297029,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52840,"kb_used_data":15192,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94319000,"statfs":{"total":96636764160,"available":96582656000,"internally_reserved":0,"allocated":15556608,"data_stored":12654702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147343,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49336,"kb_used_data":17636,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94322504,"statfs":{"total":96636764160,"available":96586244096,"internally_reserved":0,"allocated":18059264,"data_stored":15164159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:11:24.729 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:11:24.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:24.840+0000 7fc162c7e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:24.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:24.856+0000 7fc162c7e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:25.026 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:11:25.027 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:11:25.045 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":904,"stamp":"2024-03-30T13:11:24.371477+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":189698068,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":400808,"kb_used_data":135452,"kb_used_omap":46,"kb_used_meta":265169,"kb_avail":754573912,"statfs":{"total":773094113280,"available":772683685888,"internally_reserved":0,"allocated":138702848,"data_stored":115610018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47716,"internal_metadata":271533468},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.029565"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":953,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-03-30T13:11:20.349576+0000","last_change":"2024-03-30T13:10:16.176071+0000","last_active":"2024-03-30T13:11:20.349576+0000","last_peered":"2024-03-30T13:11:20.349576+0000","last_clean":"2024-03-30T13:11:20.349576+0000","last_became_active":"2024-03-30T13:09:27.168768+0000","last_became_peered":"2024-03-30T13:09:27.168768+0000","last_unstale":"2024-03-30T13:11:20.349576+0000","last_undegraded":"2024-03-30T13:11:20.349576+0000","last_fullsized":"2024-03-30T13:11:20.349576+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:10:16.175659+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:10:16.175659+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:39.530667+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":954,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-30T13:11:21.345954+0000","last_change":"2024-03-30T13:11:09.207574+0000","last_active":"2024-03-30T13:11:21.345954+0000","last_peered":"2024-03-30T13:11:21.345954+0000","last_clean":"2024-03-30T13:11:21.345954+0000","last_became_active":"2024-03-30T13:11:09.206259+0000","last_became_peered":"2024-03-30T13:11:09.206259+0000","last_unstale":"2024-03-30T13:11:21.345954+0000","last_undegraded":"2024-03-30T13:11:21.345954+0000","last_fullsized":"2024-03-30T13:11:21.345954+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_clean_scrub_stamp":"2024-03-30T13:10:44.578038+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:10.667889+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10621892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":500,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-30T13:11:22.362548+0000","last_change":"2024-03-30T13:11:09.210942+0000","last_active":"2024-03-30T13:11:22.362548+0000","last_peered":"2024-03-30T13:11:22.362548+0000","last_clean":"2024-03-30T13:11:22.362548+0000","last_became_active":"2024-03-30T13:11:09.209738+0000","last_became_peered":"2024-03-30T13:11:09.209738+0000","last_unstale":"2024-03-30T13:11:22.362548+0000","last_undegraded":"2024-03-30T13:11:22.362548+0000","last_fullsized":"2024-03-30T13:11:22.362548+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:10:54.353314+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:10:54.353314+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:58.747219+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":482,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-30T13:11:22.362167+0000","last_change":"2024-03-30T13:11:10.457971+0000","last_active":"2024-03-30T13:11:22.362167+0000","last_peered":"2024-03-30T13:11:22.362167+0000","last_clean":"2024-03-30T13:11:22.362167+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:11:22.362167+0000","last_undegraded":"2024-03-30T13:11:22.362167+0000","last_fullsized":"2024-03-30T13:11:22.362167+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:10.457629+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:11:10.457629+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:22.474745+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":303,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-30T13:11:21.405707+0000","last_change":"2024-03-30T13:11:02.527083+0000","last_active":"2024-03-30T13:11:21.405707+0000","last_peered":"2024-03-30T13:11:21.405707+0000","last_clean":"2024-03-30T13:11:21.405707+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:11:21.405707+0000","last_undegraded":"2024-03-30T13:11:21.405707+0000","last_fullsized":"2024-03-30T13:11:21.405707+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:11:02.526699+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:11:02.526699+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:19.924188+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":965,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-30T13:11:21.345506+0000","last_change":"2024-03-30T13:11:12.454138+0000","last_active":"2024-03-30T13:11:21.345506+0000","last_peered":"2024-03-30T13:11:21.345506+0000","last_clean":"2024-03-30T13:11:21.345506+0000","last_became_active":"2024-03-30T13:10:18.181149+0000","last_became_peered":"2024-03-30T13:10:18.181149+0000","last_unstale":"2024-03-30T13:11:21.345506+0000","last_undegraded":"2024-03-30T13:11:21.345506+0000","last_fullsized":"2024-03-30T13:11:21.345506+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:11:12.454045+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_clean_scrub_stamp":"2024-03-30T13:11:12.454045+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:14.977683+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":491,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-30T13:11:22.362199+0000","last_change":"2024-03-30T13:11:09.314561+0000","last_active":"2024-03-30T13:11:22.362199+0000","last_peered":"2024-03-30T13:11:22.362199+0000","last_clean":"2024-03-30T13:11:22.362199+0000","last_became_active":"2024-03-30T13:11:09.214741+0000","last_became_peered":"2024-03-30T13:11:09.214741+0000","last_unstale":"2024-03-30T13:11:22.362199+0000","last_undegraded":"2024-03-30T13:11:22.362199+0000","last_fullsized":"2024-03-30T13:11:22.362199+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:11:03.485902+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:11:03.485902+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:27.208222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":257,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-30T13:11:19.320177+0000","last_change":"2024-03-30T13:10:23.571319+0000","last_active":"2024-03-30T13:11:19.320177+0000","last_peered":"2024-03-30T13:11:19.320177+0000","last_clean":"2024-03-30T13:11:19.320177+0000","last_became_active":"2024-03-30T13:09:27.546829+0000","last_became_peered":"2024-03-30T13:09:27.546829+0000","last_unstale":"2024-03-30T13:11:19.320177+0000","last_undegraded":"2024-03-30T13:11:19.320177+0000","last_fullsized":"2024-03-30T13:11:19.320177+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:10:23.570911+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:10:23.570911+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:45.997399+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1083,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-30T13:11:21.345341+0000","last_change":"2024-03-30T13:11:09.213327+0000","last_active":"2024-03-30T13:11:21.345341+0000","last_peered":"2024-03-30T13:11:21.345341+0000","last_clean":"2024-03-30T13:11:21.345341+0000","last_became_active":"2024-03-30T13:11:09.212712+0000","last_became_peered":"2024-03-30T13:11:09.212712+0000","last_unstale":"2024-03-30T13:11:21.345341+0000","last_undegraded":"2024-03-30T13:11:21.345341+0000","last_fullsized":"2024-03-30T13:11:21.345341+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:10:52.360707+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:54.603559+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":55832128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":842,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-30T13:11:21.345943+0000","last_change":"2024-03-30T13:10:41.530383+0000","last_active":"2024-03-30T13:11:21.345943+0000","last_peered":"2024-03-30T13:11:21.345943+0000","last_clean":"2024-03-30T13:11:21.345943+0000","last_became_active":"2024-03-30T13:09:27.169549+0000","last_became_peered":"2024-03-30T13:09:27.169549+0000","last_unstale":"2024-03-30T13:11:21.345943+0000","last_undegraded":"2024-03-30T13:11:21.345943+0000","last_fullsized":"2024-03-30T13:11:21.345943+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:10:41.529990+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:10:41.529990+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:04.044498+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":968,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-03-30T13:11:20.349321+0000","last_change":"2024-03-30T13:11:20.346373+0000","last_active":"2024-03-30T13:11:20.349321+0000","last_peered":"2024-03-30T13:11:20.349321+0000","last_clean":"2024-03-30T13:11:20.349321+0000","last_became_active":"2024-03-30T13:09:27.167070+0000","last_became_peered":"2024-03-30T13:09:27.167070+0000","last_unstale":"2024-03-30T13:11:20.349321+0000","last_undegraded":"2024-03-30T13:11:20.349321+0000","last_fullsized":"2024-03-30T13:11:20.349321+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:11:20.345991+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:11:20.345991+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:24.270128+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1033,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-03-30T13:11:20.349343+0000","last_change":"2024-03-30T13:11:13.269830+0000","last_active":"2024-03-30T13:11:20.349343+0000","last_peered":"2024-03-30T13:11:20.349343+0000","last_clean":"2024-03-30T13:11:20.349343+0000","last_became_active":"2024-03-30T13:10:18.180670+0000","last_became_peered":"2024-03-30T13:10:18.180670+0000","last_unstale":"2024-03-30T13:11:20.349343+0000","last_undegraded":"2024-03-30T13:11:20.349343+0000","last_fullsized":"2024-03-30T13:11:20.349343+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:11:13.269449+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:11:13.269449+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:32.598288+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":333,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-30T13:11:21.405683+0000","last_change":"2024-03-30T13:10:18.185077+0000","last_active":"2024-03-30T13:11:21.405683+0000","last_peered":"2024-03-30T13:11:21.405683+0000","last_clean":"2024-03-30T13:11:21.405683+0000","last_became_active":"2024-03-30T13:10:18.183814+0000","last_became_peered":"2024-03-30T13:10:18.183814+0000","last_unstale":"2024-03-30T13:11:21.405683+0000","last_undegraded":"2024-03-30T13:11:21.405683+0000","last_fullsized":"2024-03-30T13:11:21.405683+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:10:13.731448+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:10:13.731448+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:23.829826+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":453,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-30T13:11:22.362191+0000","last_change":"2024-03-30T13:11:15.422620+0000","last_active":"2024-03-30T13:11:22.362191+0000","last_peered":"2024-03-30T13:11:22.362191+0000","last_clean":"2024-03-30T13:11:22.362191+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:11:22.362191+0000","last_undegraded":"2024-03-30T13:11:22.362191+0000","last_fullsized":"2024-03-30T13:11:22.362191+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:11:15.422350+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:11:15.422350+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:17.684567+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":341,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-30T13:11:21.405724+0000","last_change":"2024-03-30T13:10:48.606161+0000","last_active":"2024-03-30T13:11:21.405724+0000","last_peered":"2024-03-30T13:11:21.405724+0000","last_clean":"2024-03-30T13:11:21.405724+0000","last_became_active":"2024-03-30T13:09:27.162655+0000","last_became_peered":"2024-03-30T13:09:27.162655+0000","last_unstale":"2024-03-30T13:11:21.405724+0000","last_undegraded":"2024-03-30T13:11:21.405724+0000","last_fullsized":"2024-03-30T13:11:21.405724+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:10:48.605866+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:55.059047+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":232,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-30T13:11:19.320192+0000","last_change":"2024-03-30T13:10:08.609586+0000","last_active":"2024-03-30T13:11:19.320192+0000","last_peered":"2024-03-30T13:11:19.320192+0000","last_clean":"2024-03-30T13:11:19.320192+0000","last_became_active":"2024-03-30T13:08:39.543465+0000","last_became_peered":"2024-03-30T13:08:39.543465+0000","last_unstale":"2024-03-30T13:11:19.320192+0000","last_undegraded":"2024-03-30T13:11:19.320192+0000","last_fullsized":"2024-03-30T13:11:19.320192+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:10:08.609336+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:10:08.609336+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":189698068,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834137,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52360,"kb_used_data":20220,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94319480,"statfs":{"total":96636764160,"available":96583147520,"internally_reserved":0,"allocated":20705280,"data_stored":17823777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534766,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57588,"kb_used_data":24988,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94314252,"statfs":{"total":96636764160,"available":96577794048,"internally_reserved":0,"allocated":25587712,"data_stored":22692607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":33350120},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824077,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53436,"kb_used_data":16680,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94318404,"statfs":{"total":96636764160,"available":96582045696,"internally_reserved":0,"allocated":17080320,"data_stored":14192016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":37609969},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509633,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42604,"kb_used_data":11424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329236,"statfs":{"total":96636764160,"available":96593137664,"internally_reserved":0,"allocated":11698176,"data_stored":8823474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":346,"seq":1486058684451,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42848,"kb_used_data":10700,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94328992,"statfs":{"total":96636764160,"available":96592887808,"internally_reserved":0,"allocated":10956800,"data_stored":8082098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6045,"internal_metadata":32893027},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509633,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49796,"kb_used_data":18612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322044,"statfs":{"total":96636764160,"available":96585773056,"internally_reserved":0,"allocated":19058688,"data_stored":16177185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297029,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52840,"kb_used_data":15192,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94319000,"statfs":{"total":96636764160,"available":96582656000,"internally_reserved":0,"allocated":15556608,"data_stored":12654702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147343,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49336,"kb_used_data":17636,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94322504,"statfs":{"total":96636764160,"available":96586244096,"internally_reserved":0,"allocated":18059264,"data_stored":15164159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:11:25.046 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:11:25.046 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:11:25.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:25.164+0000 7f58e7dc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:25.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:25.176+0000 7f58e7dc5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:25.349 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:11:25.349 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":435,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:11:24.368111+0000","last_up_change":"2024-03-30T13:11:08.188580+0000","last_in_change":"2024-03-30T13:11:08.188580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":23,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":410,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":83,"last_clean_end":336,"up_from":346,"up_thru":411,"down_at":338,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2287832000},{"type":"v1","addr":"172.21.15.67:6818","nonce":2287832000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2287832000},{"type":"v1","addr":"172.21.15.67:6821","nonce":2287832000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2287832000},{"type":"v1","addr":"172.21.15.67:6828","nonce":2287832000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2287832000},{"type":"v1","addr":"172.21.15.67:6825","nonce":2287832000}]},"public_addr":"172.21.15.67:6818/2287832000","cluster_addr":"172.21.15.67:6821/2287832000","heartbeat_back_addr":"172.21.15.67:6828/2287832000","heartbeat_front_addr":"172.21.15.67:6825/2287832000","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":411,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":388,"up_from":393,"up_thru":419,"down_at":389,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6801","nonce":3837194718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6803","nonce":3837194718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6807","nonce":3837194718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6805","nonce":3837194718}]},"public_addr":"172.21.15.67:6801/3837194718","cluster_addr":"172.21.15.67:6803/3837194718","heartbeat_back_addr":"172.21.15.67:6807/3837194718","heartbeat_front_addr":"172.21.15.67:6805/3837194718","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":135,"last_clean_end":243,"up_from":250,"up_thru":419,"down_at":244,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6803","nonce":3986522908}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6807","nonce":3986522908}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6815","nonce":3986522908}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":3986522908},{"type":"v1","addr":"172.21.15.82:6811","nonce":3986522908}]},"public_addr":"172.21.15.82:6803/3986522908","cluster_addr":"172.21.15.82:6807/3986522908","heartbeat_back_addr":"172.21.15.82:6815/3986522908","heartbeat_front_addr":"172.21.15.82:6811/3986522908","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":274,"last_clean_end":314,"up_from":320,"up_thru":411,"down_at":315,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":925466951},{"type":"v1","addr":"172.21.15.82:6817","nonce":925466951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":925466951},{"type":"v1","addr":"172.21.15.82:6819","nonce":925466951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":925466951},{"type":"v1","addr":"172.21.15.82:6823","nonce":925466951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":925466951},{"type":"v1","addr":"172.21.15.82:6821","nonce":925466951}]},"public_addr":"172.21.15.82:6817/925466951","cluster_addr":"172.21.15.82:6819/925466951","heartbeat_back_addr":"172.21.15.82:6823/925466951","heartbeat_front_addr":"172.21.15.82:6821/925466951","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":227,"last_clean_end":363,"up_from":372,"up_thru":393,"down_at":364,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6801","nonce":189641108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6806","nonce":189641108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6814","nonce":189641108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6810","nonce":189641108}]},"public_addr":"172.21.15.82:6801/189641108","cluster_addr":"172.21.15.82:6806/189641108","heartbeat_back_addr":"172.21.15.82:6814/189641108","heartbeat_front_addr":"172.21.15.82:6810/189641108","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":298,"last_clean_end":409,"up_from":419,"up_thru":419,"down_at":410,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6825","nonce":2166599911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6827","nonce":2166599911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6831","nonce":2166599911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6829","nonce":2166599911}]},"public_addr":"172.21.15.82:6825/2166599911","cluster_addr":"172.21.15.82:6827/2166599911","heartbeat_back_addr":"172.21.15.82:6831/2166599911","heartbeat_front_addr":"172.21.15.82:6829/2166599911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:10:40.093927+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:10:58.695970+0000","dead_epoch":338},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:10:39.133828+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:10:16.338119+0000","dead_epoch":389},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:10:44.037376+0000","dead_epoch":244},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:10:52.682027+0000","dead_epoch":315},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:10:36.919162+0000","dead_epoch":364},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:10:34.551034+0000","dead_epoch":410}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:11:25.368 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:11:25.368 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:11:25.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:25.484+0000 7f716a015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:25.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:25.504+0000 7f716a015640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:25.677 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:11:25.696 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:11:25.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:25.812+0000 7fde2126b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:25.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:25.828+0000 7fde2126b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:26.011 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:11:26.011 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:11:26.030 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:11:26.031 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:11:26.031 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:11:26.031 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:11:26.031 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 4, 5, 1, 6, 3, 7] 2024-03-30T13:11:26.031 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:11:26.031 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:11:26.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:26.144+0000 7fa5090bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:26.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:26.156+0000 7fa5090bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:26.331 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:11:26.332 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:11:26.348 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":905,"stamp":"2024-03-30T13:11:26.056399+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":189698068,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":400988,"kb_used_data":135632,"kb_used_omap":46,"kb_used_meta":265169,"kb_avail":754573732,"statfs":{"total":773094113280,"available":772683501568,"internally_reserved":0,"allocated":138887168,"data_stored":115692323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47716,"internal_metadata":271533468},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.713463"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":953,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-03-30T13:11:20.349576+0000","last_change":"2024-03-30T13:10:16.176071+0000","last_active":"2024-03-30T13:11:20.349576+0000","last_peered":"2024-03-30T13:11:20.349576+0000","last_clean":"2024-03-30T13:11:20.349576+0000","last_became_active":"2024-03-30T13:09:27.168768+0000","last_became_peered":"2024-03-30T13:09:27.168768+0000","last_unstale":"2024-03-30T13:11:20.349576+0000","last_undegraded":"2024-03-30T13:11:20.349576+0000","last_fullsized":"2024-03-30T13:11:20.349576+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:10:16.175659+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:10:16.175659+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:39.530667+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":954,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-30T13:11:21.345954+0000","last_change":"2024-03-30T13:11:09.207574+0000","last_active":"2024-03-30T13:11:21.345954+0000","last_peered":"2024-03-30T13:11:21.345954+0000","last_clean":"2024-03-30T13:11:21.345954+0000","last_became_active":"2024-03-30T13:11:09.206259+0000","last_became_peered":"2024-03-30T13:11:09.206259+0000","last_unstale":"2024-03-30T13:11:21.345954+0000","last_undegraded":"2024-03-30T13:11:21.345954+0000","last_fullsized":"2024-03-30T13:11:21.345954+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_clean_scrub_stamp":"2024-03-30T13:10:44.578038+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:10.667889+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10621892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":500,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-30T13:11:22.362548+0000","last_change":"2024-03-30T13:11:09.210942+0000","last_active":"2024-03-30T13:11:22.362548+0000","last_peered":"2024-03-30T13:11:22.362548+0000","last_clean":"2024-03-30T13:11:22.362548+0000","last_became_active":"2024-03-30T13:11:09.209738+0000","last_became_peered":"2024-03-30T13:11:09.209738+0000","last_unstale":"2024-03-30T13:11:22.362548+0000","last_undegraded":"2024-03-30T13:11:22.362548+0000","last_fullsized":"2024-03-30T13:11:22.362548+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:10:54.353314+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:10:54.353314+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:58.747219+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":482,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-30T13:11:22.362167+0000","last_change":"2024-03-30T13:11:10.457971+0000","last_active":"2024-03-30T13:11:22.362167+0000","last_peered":"2024-03-30T13:11:22.362167+0000","last_clean":"2024-03-30T13:11:22.362167+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:11:22.362167+0000","last_undegraded":"2024-03-30T13:11:22.362167+0000","last_fullsized":"2024-03-30T13:11:22.362167+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:10.457629+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:11:10.457629+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:22.474745+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":303,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-30T13:11:21.405707+0000","last_change":"2024-03-30T13:11:02.527083+0000","last_active":"2024-03-30T13:11:21.405707+0000","last_peered":"2024-03-30T13:11:21.405707+0000","last_clean":"2024-03-30T13:11:21.405707+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:11:21.405707+0000","last_undegraded":"2024-03-30T13:11:21.405707+0000","last_fullsized":"2024-03-30T13:11:21.405707+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:11:02.526699+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:11:02.526699+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:19.924188+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":965,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-30T13:11:21.345506+0000","last_change":"2024-03-30T13:11:12.454138+0000","last_active":"2024-03-30T13:11:21.345506+0000","last_peered":"2024-03-30T13:11:21.345506+0000","last_clean":"2024-03-30T13:11:21.345506+0000","last_became_active":"2024-03-30T13:10:18.181149+0000","last_became_peered":"2024-03-30T13:10:18.181149+0000","last_unstale":"2024-03-30T13:11:21.345506+0000","last_undegraded":"2024-03-30T13:11:21.345506+0000","last_fullsized":"2024-03-30T13:11:21.345506+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:11:12.454045+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_clean_scrub_stamp":"2024-03-30T13:11:12.454045+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:14.977683+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":491,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-30T13:11:22.362199+0000","last_change":"2024-03-30T13:11:09.314561+0000","last_active":"2024-03-30T13:11:22.362199+0000","last_peered":"2024-03-30T13:11:22.362199+0000","last_clean":"2024-03-30T13:11:22.362199+0000","last_became_active":"2024-03-30T13:11:09.214741+0000","last_became_peered":"2024-03-30T13:11:09.214741+0000","last_unstale":"2024-03-30T13:11:22.362199+0000","last_undegraded":"2024-03-30T13:11:22.362199+0000","last_fullsized":"2024-03-30T13:11:22.362199+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:11:03.485902+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:11:03.485902+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:27.208222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":267,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-30T13:11:24.373844+0000","last_change":"2024-03-30T13:10:23.571319+0000","last_active":"2024-03-30T13:11:24.373844+0000","last_peered":"2024-03-30T13:11:24.373844+0000","last_clean":"2024-03-30T13:11:24.373844+0000","last_became_active":"2024-03-30T13:09:27.546829+0000","last_became_peered":"2024-03-30T13:09:27.546829+0000","last_unstale":"2024-03-30T13:11:24.373844+0000","last_undegraded":"2024-03-30T13:11:24.373844+0000","last_fullsized":"2024-03-30T13:11:24.373844+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:10:23.570911+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:10:23.570911+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:45.997399+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1083,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-30T13:11:21.345341+0000","last_change":"2024-03-30T13:11:09.213327+0000","last_active":"2024-03-30T13:11:21.345341+0000","last_peered":"2024-03-30T13:11:21.345341+0000","last_clean":"2024-03-30T13:11:21.345341+0000","last_became_active":"2024-03-30T13:11:09.212712+0000","last_became_peered":"2024-03-30T13:11:09.212712+0000","last_unstale":"2024-03-30T13:11:21.345341+0000","last_undegraded":"2024-03-30T13:11:21.345341+0000","last_fullsized":"2024-03-30T13:11:21.345341+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:10:52.360707+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:54.603559+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":55832128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":842,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-30T13:11:21.345943+0000","last_change":"2024-03-30T13:10:41.530383+0000","last_active":"2024-03-30T13:11:21.345943+0000","last_peered":"2024-03-30T13:11:21.345943+0000","last_clean":"2024-03-30T13:11:21.345943+0000","last_became_active":"2024-03-30T13:09:27.169549+0000","last_became_peered":"2024-03-30T13:09:27.169549+0000","last_unstale":"2024-03-30T13:11:21.345943+0000","last_undegraded":"2024-03-30T13:11:21.345943+0000","last_fullsized":"2024-03-30T13:11:21.345943+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:10:41.529990+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:10:41.529990+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:04.044498+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":968,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-03-30T13:11:20.349321+0000","last_change":"2024-03-30T13:11:20.346373+0000","last_active":"2024-03-30T13:11:20.349321+0000","last_peered":"2024-03-30T13:11:20.349321+0000","last_clean":"2024-03-30T13:11:20.349321+0000","last_became_active":"2024-03-30T13:09:27.167070+0000","last_became_peered":"2024-03-30T13:09:27.167070+0000","last_unstale":"2024-03-30T13:11:20.349321+0000","last_undegraded":"2024-03-30T13:11:20.349321+0000","last_fullsized":"2024-03-30T13:11:20.349321+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:11:20.345991+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:11:20.345991+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:24.270128+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1033,"reported_epoch":431,"state":"active+clean","last_fresh":"2024-03-30T13:11:20.349343+0000","last_change":"2024-03-30T13:11:13.269830+0000","last_active":"2024-03-30T13:11:20.349343+0000","last_peered":"2024-03-30T13:11:20.349343+0000","last_clean":"2024-03-30T13:11:20.349343+0000","last_became_active":"2024-03-30T13:10:18.180670+0000","last_became_peered":"2024-03-30T13:10:18.180670+0000","last_unstale":"2024-03-30T13:11:20.349343+0000","last_undegraded":"2024-03-30T13:11:20.349343+0000","last_fullsized":"2024-03-30T13:11:20.349343+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:11:13.269449+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:11:13.269449+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:32.598288+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":333,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-30T13:11:21.405683+0000","last_change":"2024-03-30T13:10:18.185077+0000","last_active":"2024-03-30T13:11:21.405683+0000","last_peered":"2024-03-30T13:11:21.405683+0000","last_clean":"2024-03-30T13:11:21.405683+0000","last_became_active":"2024-03-30T13:10:18.183814+0000","last_became_peered":"2024-03-30T13:10:18.183814+0000","last_unstale":"2024-03-30T13:11:21.405683+0000","last_undegraded":"2024-03-30T13:11:21.405683+0000","last_fullsized":"2024-03-30T13:11:21.405683+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:10:13.731448+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:10:13.731448+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:23.829826+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":453,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-30T13:11:22.362191+0000","last_change":"2024-03-30T13:11:15.422620+0000","last_active":"2024-03-30T13:11:22.362191+0000","last_peered":"2024-03-30T13:11:22.362191+0000","last_clean":"2024-03-30T13:11:22.362191+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:11:22.362191+0000","last_undegraded":"2024-03-30T13:11:22.362191+0000","last_fullsized":"2024-03-30T13:11:22.362191+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:11:15.422350+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:11:15.422350+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:17.684567+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":341,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-30T13:11:21.405724+0000","last_change":"2024-03-30T13:10:48.606161+0000","last_active":"2024-03-30T13:11:21.405724+0000","last_peered":"2024-03-30T13:11:21.405724+0000","last_clean":"2024-03-30T13:11:21.405724+0000","last_became_active":"2024-03-30T13:09:27.162655+0000","last_became_peered":"2024-03-30T13:09:27.162655+0000","last_unstale":"2024-03-30T13:11:21.405724+0000","last_undegraded":"2024-03-30T13:11:21.405724+0000","last_fullsized":"2024-03-30T13:11:21.405724+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:10:48.605866+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:55.059047+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":250,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-30T13:11:24.373858+0000","last_change":"2024-03-30T13:11:19.631799+0000","last_active":"2024-03-30T13:11:24.373858+0000","last_peered":"2024-03-30T13:11:24.373858+0000","last_clean":"2024-03-30T13:11:24.373858+0000","last_became_active":"2024-03-30T13:08:39.543465+0000","last_became_peered":"2024-03-30T13:08:39.543465+0000","last_unstale":"2024-03-30T13:11:24.373858+0000","last_undegraded":"2024-03-30T13:11:24.373858+0000","last_fullsized":"2024-03-30T13:11:24.373858+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:19.631559+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:11:19.631559+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:29.111571+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":189698068,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834137,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52360,"kb_used_data":20220,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94319480,"statfs":{"total":96636764160,"available":96583147520,"internally_reserved":0,"allocated":20705280,"data_stored":17823777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534766,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57588,"kb_used_data":24988,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94314252,"statfs":{"total":96636764160,"available":96577794048,"internally_reserved":0,"allocated":25587712,"data_stored":22692607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":33350120},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":250,"seq":1073741824077,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53436,"kb_used_data":16680,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94318404,"statfs":{"total":96636764160,"available":96582045696,"internally_reserved":0,"allocated":17080320,"data_stored":14192016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7695,"internal_metadata":37609969},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509634,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42668,"kb_used_data":11484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329172,"statfs":{"total":96636764160,"available":96593072128,"internally_reserved":0,"allocated":11759616,"data_stored":8850909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":346,"seq":1486058684452,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42904,"kb_used_data":10760,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94328936,"statfs":{"total":96636764160,"available":96592830464,"internally_reserved":0,"allocated":11018240,"data_stored":8109533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6045,"internal_metadata":32893027},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509634,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49856,"kb_used_data":18672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321984,"statfs":{"total":96636764160,"available":96585711616,"internally_reserved":0,"allocated":19120128,"data_stored":16204620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297029,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52840,"kb_used_data":15192,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94319000,"statfs":{"total":96636764160,"available":96582656000,"internally_reserved":0,"allocated":15556608,"data_stored":12654702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147343,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49336,"kb_used_data":17636,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94322504,"statfs":{"total":96636764160,"available":96586244096,"internally_reserved":0,"allocated":18059264,"data_stored":15164159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:11:26.349 INFO:tasks.thrashosds.thrasher:Choosing PG 1.5 with acting set [4, 2, 6] 2024-03-30T13:11:26.349 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [0, 2, 4, 5, 1, 6, 3, 7] 2024-03-30T13:11:26.349 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-30T13:11:26.349 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:11:26.411 INFO:tasks.ceph.osd.4:Stopped 2024-03-30T13:11:26.412 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 2, 4, 5, 1, 6, 3, 7] 2024-03-30T13:11:26.412 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 4 2024-03-30T13:11:26.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:26.524+0000 7fd398694640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:26.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:26.536+0000 7fd398694640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:26.716 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:11:26.716+0000 7fbb38f08640 -1 mon.a@0(leader).osd e435 definitely_dead 0 2024-03-30T13:11:27.384 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.4. 2024-03-30T13:11:27.413 INFO:tasks.thrashosds.thrasher:dead_osds=[4], live_osds=[0, 2, 5, 1, 6, 3, 7] 2024-03-30T13:11:27.413 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:11:27.413 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:11:27.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:27.524+0000 7fa85f89a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:27.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:27.536+0000 7fa85f89a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:27.710 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:11:27.710 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:11:27.728 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":906,"stamp":"2024-03-30T13:11:27.388375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":189698068,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":347640,"kb_used_data":119048,"kb_used_omap":39,"kb_used_meta":228440,"kb_avail":660255240,"statfs":{"total":676457349120,"available":676101365760,"internally_reserved":0,"allocated":121905152,"data_stored":101544203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40021,"internal_metadata":233923499},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.333037"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":959,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-30T13:11:24.699515+0000","last_change":"2024-03-30T13:10:16.176071+0000","last_active":"2024-03-30T13:11:24.699515+0000","last_peered":"2024-03-30T13:11:24.699515+0000","last_clean":"2024-03-30T13:11:24.699515+0000","last_became_active":"2024-03-30T13:09:27.168768+0000","last_became_peered":"2024-03-30T13:09:27.168768+0000","last_unstale":"2024-03-30T13:11:24.699515+0000","last_undegraded":"2024-03-30T13:11:24.699515+0000","last_fullsized":"2024-03-30T13:11:24.699515+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:10:16.175659+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:10:16.175659+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:39.530667+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":954,"reported_epoch":432,"state":"stale+active+clean","last_fresh":"2024-03-30T13:11:21.345954+0000","last_change":"2024-03-30T13:11:09.207574+0000","last_active":"2024-03-30T13:11:21.345954+0000","last_peered":"2024-03-30T13:11:21.345954+0000","last_clean":"2024-03-30T13:11:21.345954+0000","last_became_active":"2024-03-30T13:11:09.206259+0000","last_became_peered":"2024-03-30T13:11:09.206259+0000","last_unstale":"2024-03-30T13:11:27.388397+0000","last_undegraded":"2024-03-30T13:11:21.345954+0000","last_fullsized":"2024-03-30T13:11:21.345954+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_clean_scrub_stamp":"2024-03-30T13:10:44.578038+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:10.667889+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10621892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":500,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-30T13:11:22.362548+0000","last_change":"2024-03-30T13:11:09.210942+0000","last_active":"2024-03-30T13:11:22.362548+0000","last_peered":"2024-03-30T13:11:22.362548+0000","last_clean":"2024-03-30T13:11:22.362548+0000","last_became_active":"2024-03-30T13:11:09.209738+0000","last_became_peered":"2024-03-30T13:11:09.209738+0000","last_unstale":"2024-03-30T13:11:22.362548+0000","last_undegraded":"2024-03-30T13:11:22.362548+0000","last_fullsized":"2024-03-30T13:11:22.362548+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:10:54.353314+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:10:54.353314+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:58.747219+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":482,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-30T13:11:22.362167+0000","last_change":"2024-03-30T13:11:10.457971+0000","last_active":"2024-03-30T13:11:22.362167+0000","last_peered":"2024-03-30T13:11:22.362167+0000","last_clean":"2024-03-30T13:11:22.362167+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:11:22.362167+0000","last_undegraded":"2024-03-30T13:11:22.362167+0000","last_fullsized":"2024-03-30T13:11:22.362167+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:10.457629+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:11:10.457629+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:22.474745+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":303,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-30T13:11:21.405707+0000","last_change":"2024-03-30T13:11:02.527083+0000","last_active":"2024-03-30T13:11:21.405707+0000","last_peered":"2024-03-30T13:11:21.405707+0000","last_clean":"2024-03-30T13:11:21.405707+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:11:21.405707+0000","last_undegraded":"2024-03-30T13:11:21.405707+0000","last_fullsized":"2024-03-30T13:11:21.405707+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:11:02.526699+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:11:02.526699+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:19.924188+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":969,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-30T13:11:24.488926+0000","last_change":"2024-03-30T13:11:12.454138+0000","last_active":"2024-03-30T13:11:24.488926+0000","last_peered":"2024-03-30T13:11:24.488926+0000","last_clean":"2024-03-30T13:11:24.488926+0000","last_became_active":"2024-03-30T13:10:18.181149+0000","last_became_peered":"2024-03-30T13:10:18.181149+0000","last_unstale":"2024-03-30T13:11:24.488926+0000","last_undegraded":"2024-03-30T13:11:24.488926+0000","last_fullsized":"2024-03-30T13:11:24.488926+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:11:12.454045+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_clean_scrub_stamp":"2024-03-30T13:11:12.454045+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:14.977683+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":491,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-30T13:11:22.362199+0000","last_change":"2024-03-30T13:11:09.314561+0000","last_active":"2024-03-30T13:11:22.362199+0000","last_peered":"2024-03-30T13:11:22.362199+0000","last_clean":"2024-03-30T13:11:22.362199+0000","last_became_active":"2024-03-30T13:11:09.214741+0000","last_became_peered":"2024-03-30T13:11:09.214741+0000","last_unstale":"2024-03-30T13:11:22.362199+0000","last_undegraded":"2024-03-30T13:11:22.362199+0000","last_fullsized":"2024-03-30T13:11:22.362199+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:11:03.485902+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:11:03.485902+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:27.208222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":267,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-30T13:11:24.373844+0000","last_change":"2024-03-30T13:10:23.571319+0000","last_active":"2024-03-30T13:11:24.373844+0000","last_peered":"2024-03-30T13:11:24.373844+0000","last_clean":"2024-03-30T13:11:24.373844+0000","last_became_active":"2024-03-30T13:09:27.546829+0000","last_became_peered":"2024-03-30T13:09:27.546829+0000","last_unstale":"2024-03-30T13:11:24.373844+0000","last_undegraded":"2024-03-30T13:11:24.373844+0000","last_fullsized":"2024-03-30T13:11:24.373844+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:10:23.570911+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:10:23.570911+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:45.997399+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1087,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-30T13:11:24.488858+0000","last_change":"2024-03-30T13:11:09.213327+0000","last_active":"2024-03-30T13:11:24.488858+0000","last_peered":"2024-03-30T13:11:24.488858+0000","last_clean":"2024-03-30T13:11:24.488858+0000","last_became_active":"2024-03-30T13:11:09.212712+0000","last_became_peered":"2024-03-30T13:11:09.212712+0000","last_unstale":"2024-03-30T13:11:24.488858+0000","last_undegraded":"2024-03-30T13:11:24.488858+0000","last_fullsized":"2024-03-30T13:11:24.488858+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:10:52.360707+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:54.603559+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":55832128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":842,"reported_epoch":432,"state":"stale+active+clean","last_fresh":"2024-03-30T13:11:21.345943+0000","last_change":"2024-03-30T13:10:41.530383+0000","last_active":"2024-03-30T13:11:21.345943+0000","last_peered":"2024-03-30T13:11:21.345943+0000","last_clean":"2024-03-30T13:11:21.345943+0000","last_became_active":"2024-03-30T13:09:27.169549+0000","last_became_peered":"2024-03-30T13:09:27.169549+0000","last_unstale":"2024-03-30T13:11:27.388400+0000","last_undegraded":"2024-03-30T13:11:21.345943+0000","last_fullsized":"2024-03-30T13:11:21.345943+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:10:41.529990+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:10:41.529990+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:04.044498+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":974,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-30T13:11:24.699405+0000","last_change":"2024-03-30T13:11:20.346373+0000","last_active":"2024-03-30T13:11:24.699405+0000","last_peered":"2024-03-30T13:11:24.699405+0000","last_clean":"2024-03-30T13:11:24.699405+0000","last_became_active":"2024-03-30T13:09:27.167070+0000","last_became_peered":"2024-03-30T13:09:27.167070+0000","last_unstale":"2024-03-30T13:11:24.699405+0000","last_undegraded":"2024-03-30T13:11:24.699405+0000","last_fullsized":"2024-03-30T13:11:24.699405+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:11:20.345991+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:11:20.345991+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:24.270128+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1039,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-30T13:11:24.699469+0000","last_change":"2024-03-30T13:11:13.269830+0000","last_active":"2024-03-30T13:11:24.699469+0000","last_peered":"2024-03-30T13:11:24.699469+0000","last_clean":"2024-03-30T13:11:24.699469+0000","last_became_active":"2024-03-30T13:10:18.180670+0000","last_became_peered":"2024-03-30T13:10:18.180670+0000","last_unstale":"2024-03-30T13:11:24.699469+0000","last_undegraded":"2024-03-30T13:11:24.699469+0000","last_fullsized":"2024-03-30T13:11:24.699469+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:11:13.269449+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:11:13.269449+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:32.598288+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":333,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-30T13:11:21.405683+0000","last_change":"2024-03-30T13:10:18.185077+0000","last_active":"2024-03-30T13:11:21.405683+0000","last_peered":"2024-03-30T13:11:21.405683+0000","last_clean":"2024-03-30T13:11:21.405683+0000","last_became_active":"2024-03-30T13:10:18.183814+0000","last_became_peered":"2024-03-30T13:10:18.183814+0000","last_unstale":"2024-03-30T13:11:21.405683+0000","last_undegraded":"2024-03-30T13:11:21.405683+0000","last_fullsized":"2024-03-30T13:11:21.405683+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:10:13.731448+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:10:13.731448+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:23.829826+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":453,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-30T13:11:22.362191+0000","last_change":"2024-03-30T13:11:15.422620+0000","last_active":"2024-03-30T13:11:22.362191+0000","last_peered":"2024-03-30T13:11:22.362191+0000","last_clean":"2024-03-30T13:11:22.362191+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:11:22.362191+0000","last_undegraded":"2024-03-30T13:11:22.362191+0000","last_fullsized":"2024-03-30T13:11:22.362191+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:11:15.422350+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:11:15.422350+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:17.684567+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":341,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-30T13:11:21.405724+0000","last_change":"2024-03-30T13:10:48.606161+0000","last_active":"2024-03-30T13:11:21.405724+0000","last_peered":"2024-03-30T13:11:21.405724+0000","last_clean":"2024-03-30T13:11:21.405724+0000","last_became_active":"2024-03-30T13:09:27.162655+0000","last_became_peered":"2024-03-30T13:09:27.162655+0000","last_unstale":"2024-03-30T13:11:21.405724+0000","last_undegraded":"2024-03-30T13:11:21.405724+0000","last_fullsized":"2024-03-30T13:11:21.405724+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:10:48.605866+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:55.059047+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":250,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-03-30T13:11:24.373858+0000","last_change":"2024-03-30T13:11:19.631799+0000","last_active":"2024-03-30T13:11:24.373858+0000","last_peered":"2024-03-30T13:11:24.373858+0000","last_clean":"2024-03-30T13:11:24.373858+0000","last_became_active":"2024-03-30T13:08:39.543465+0000","last_became_peered":"2024-03-30T13:08:39.543465+0000","last_unstale":"2024-03-30T13:11:24.373858+0000","last_undegraded":"2024-03-30T13:11:24.373858+0000","last_fullsized":"2024-03-30T13:11:24.373858+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:19.631559+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:11:19.631559+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:29.111571+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":189698068,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834138,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52420,"kb_used_data":20280,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94319420,"statfs":{"total":96636764160,"available":96583086080,"internally_reserved":0,"allocated":20766720,"data_stored":17851212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534766,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57588,"kb_used_data":24988,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94314252,"statfs":{"total":96636764160,"available":96577794048,"internally_reserved":0,"allocated":25587712,"data_stored":22692607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":33350120},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509634,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42668,"kb_used_data":11484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329172,"statfs":{"total":96636764160,"available":96593072128,"internally_reserved":0,"allocated":11759616,"data_stored":8850909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":346,"seq":1486058684452,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42904,"kb_used_data":10760,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94328936,"statfs":{"total":96636764160,"available":96592830464,"internally_reserved":0,"allocated":11018240,"data_stored":8109533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6045,"internal_metadata":32893027},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509634,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49856,"kb_used_data":18672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321984,"statfs":{"total":96636764160,"available":96585711616,"internally_reserved":0,"allocated":19120128,"data_stored":16204620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297029,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52840,"kb_used_data":15192,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94319000,"statfs":{"total":96636764160,"available":96582656000,"internally_reserved":0,"allocated":15556608,"data_stored":12654702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147344,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49364,"kb_used_data":17672,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94322476,"statfs":{"total":96636764160,"available":96586215424,"internally_reserved":0,"allocated":18096128,"data_stored":15180620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:11:27.729 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:11:52.730 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:11:52.730 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:11:52.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:52.855+0000 7fb28e3de640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:52.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:52.867+0000 7fb28e3de640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:53.045 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:11:53.045 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:11:53.066 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":922,"stamp":"2024-03-30T13:11:52.064605+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":219177150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":360208,"kb_used_data":131644,"kb_used_omap":39,"kb_used_meta":228440,"kb_avail":660242672,"statfs":{"total":676457349120,"available":676088496128,"internally_reserved":0,"allocated":134803456,"data_stored":114226793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40021,"internal_metadata":233923499},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002244"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":988,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-30T13:11:49.302365+0000","last_change":"2024-03-30T13:11:37.448160+0000","last_active":"2024-03-30T13:11:49.302365+0000","last_peered":"2024-03-30T13:11:49.302365+0000","last_clean":"2024-03-30T13:11:49.302365+0000","last_became_active":"2024-03-30T13:11:28.420293+0000","last_became_peered":"2024-03-30T13:11:28.420293+0000","last_unstale":"2024-03-30T13:11:49.302365+0000","last_undegraded":"2024-03-30T13:11:49.302365+0000","last_fullsized":"2024-03-30T13:11:49.302365+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:37.447802+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:11:37.447802+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:40.662880+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":19,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-30T13:11:49.301993+0000","last_change":"2024-03-30T13:11:28.451217+0000","last_active":"2024-03-30T13:11:49.301993+0000","last_peered":"2024-03-30T13:11:49.301993+0000","last_clean":"2024-03-30T13:11:49.301993+0000","last_became_active":"2024-03-30T13:11:28.421016+0000","last_became_peered":"2024-03-30T13:11:28.421016+0000","last_unstale":"2024-03-30T13:11:49.301993+0000","last_undegraded":"2024-03-30T13:11:49.301993+0000","last_fullsized":"2024-03-30T13:11:49.301993+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_clean_scrub_stamp":"2024-03-30T13:10:44.578038+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:12.834694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":524,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-30T13:11:44.601363+0000","last_change":"2024-03-30T13:11:28.487913+0000","last_active":"2024-03-30T13:11:44.601363+0000","last_peered":"2024-03-30T13:11:44.601363+0000","last_clean":"2024-03-30T13:11:44.601363+0000","last_became_active":"2024-03-30T13:11:28.415706+0000","last_became_peered":"2024-03-30T13:11:28.415706+0000","last_unstale":"2024-03-30T13:11:44.601363+0000","last_undegraded":"2024-03-30T13:11:44.601363+0000","last_fullsized":"2024-03-30T13:11:44.601363+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:10:54.353314+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:10:54.353314+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:19.164013+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":492,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-30T13:11:44.601255+0000","last_change":"2024-03-30T13:11:10.457971+0000","last_active":"2024-03-30T13:11:44.601255+0000","last_peered":"2024-03-30T13:11:44.601255+0000","last_clean":"2024-03-30T13:11:44.601255+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:11:44.601255+0000","last_undegraded":"2024-03-30T13:11:44.601255+0000","last_fullsized":"2024-03-30T13:11:44.601255+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:10.457629+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:11:10.457629+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:22.474745+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":315,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-30T13:11:47.462166+0000","last_change":"2024-03-30T13:11:02.527083+0000","last_active":"2024-03-30T13:11:47.462166+0000","last_peered":"2024-03-30T13:11:47.462166+0000","last_clean":"2024-03-30T13:11:47.462166+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:11:47.462166+0000","last_undegraded":"2024-03-30T13:11:47.462166+0000","last_fullsized":"2024-03-30T13:11:47.462166+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:11:02.526699+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:11:02.526699+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:19.924188+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":975,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-30T13:11:44.602152+0000","last_change":"2024-03-30T13:11:12.454138+0000","last_active":"2024-03-30T13:11:44.602152+0000","last_peered":"2024-03-30T13:11:44.602152+0000","last_clean":"2024-03-30T13:11:44.602152+0000","last_became_active":"2024-03-30T13:10:18.181149+0000","last_became_peered":"2024-03-30T13:10:18.181149+0000","last_unstale":"2024-03-30T13:11:44.602152+0000","last_undegraded":"2024-03-30T13:11:44.602152+0000","last_fullsized":"2024-03-30T13:11:44.602152+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:11:12.454045+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_clean_scrub_stamp":"2024-03-30T13:11:12.454045+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:14.977683+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":521,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-30T13:11:44.601301+0000","last_change":"2024-03-30T13:11:28.555777+0000","last_active":"2024-03-30T13:11:44.601301+0000","last_peered":"2024-03-30T13:11:44.601301+0000","last_clean":"2024-03-30T13:11:44.601301+0000","last_became_active":"2024-03-30T13:11:28.418296+0000","last_became_peered":"2024-03-30T13:11:28.418296+0000","last_unstale":"2024-03-30T13:11:44.601301+0000","last_undegraded":"2024-03-30T13:11:44.601301+0000","last_fullsized":"2024-03-30T13:11:44.601301+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:11:03.485902+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:11:03.485902+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:04.574252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":275,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-30T13:11:47.462903+0000","last_change":"2024-03-30T13:10:23.571319+0000","last_active":"2024-03-30T13:11:47.462903+0000","last_peered":"2024-03-30T13:11:47.462903+0000","last_clean":"2024-03-30T13:11:47.462903+0000","last_became_active":"2024-03-30T13:09:27.546829+0000","last_became_peered":"2024-03-30T13:09:27.546829+0000","last_unstale":"2024-03-30T13:11:47.462903+0000","last_undegraded":"2024-03-30T13:11:47.462903+0000","last_fullsized":"2024-03-30T13:11:47.462903+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:10:23.570911+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:10:23.570911+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1093,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-30T13:11:44.601920+0000","last_change":"2024-03-30T13:11:09.213327+0000","last_active":"2024-03-30T13:11:44.601920+0000","last_peered":"2024-03-30T13:11:44.601920+0000","last_clean":"2024-03-30T13:11:44.601920+0000","last_became_active":"2024-03-30T13:11:09.212712+0000","last_became_peered":"2024-03-30T13:11:09.212712+0000","last_unstale":"2024-03-30T13:11:44.601920+0000","last_undegraded":"2024-03-30T13:11:44.601920+0000","last_fullsized":"2024-03-30T13:11:44.601920+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:10:52.360707+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:54.603559+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":55832128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":19,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-30T13:11:48.444996+0000","last_change":"2024-03-30T13:11:28.448070+0000","last_active":"2024-03-30T13:11:48.444996+0000","last_peered":"2024-03-30T13:11:48.444996+0000","last_clean":"2024-03-30T13:11:48.444996+0000","last_became_active":"2024-03-30T13:11:28.418790+0000","last_became_peered":"2024-03-30T13:11:28.418790+0000","last_unstale":"2024-03-30T13:11:48.444996+0000","last_undegraded":"2024-03-30T13:11:48.444996+0000","last_fullsized":"2024-03-30T13:11:48.444996+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:10:41.529990+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:10:41.529990+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:04.883022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1001,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-30T13:11:49.302213+0000","last_change":"2024-03-30T13:11:29.514209+0000","last_active":"2024-03-30T13:11:49.302213+0000","last_peered":"2024-03-30T13:11:49.302213+0000","last_clean":"2024-03-30T13:11:49.302213+0000","last_became_active":"2024-03-30T13:11:28.422548+0000","last_became_peered":"2024-03-30T13:11:28.422548+0000","last_unstale":"2024-03-30T13:11:49.302213+0000","last_undegraded":"2024-03-30T13:11:49.302213+0000","last_fullsized":"2024-03-30T13:11:49.302213+0000","mapping_epoch":436,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":437,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:11:20.345991+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:11:20.345991+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:26.014508+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1048,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-30T13:11:49.302392+0000","last_change":"2024-03-30T13:11:13.269830+0000","last_active":"2024-03-30T13:11:49.302392+0000","last_peered":"2024-03-30T13:11:49.302392+0000","last_clean":"2024-03-30T13:11:49.302392+0000","last_became_active":"2024-03-30T13:10:18.180670+0000","last_became_peered":"2024-03-30T13:10:18.180670+0000","last_unstale":"2024-03-30T13:11:49.302392+0000","last_undegraded":"2024-03-30T13:11:49.302392+0000","last_fullsized":"2024-03-30T13:11:49.302392+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:11:13.269449+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:11:13.269449+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:32.598288+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":353,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-30T13:11:47.462155+0000","last_change":"2024-03-30T13:11:24.587378+0000","last_active":"2024-03-30T13:11:47.462155+0000","last_peered":"2024-03-30T13:11:47.462155+0000","last_clean":"2024-03-30T13:11:47.462155+0000","last_became_active":"2024-03-30T13:10:18.183814+0000","last_became_peered":"2024-03-30T13:10:18.183814+0000","last_unstale":"2024-03-30T13:11:47.462155+0000","last_undegraded":"2024-03-30T13:11:47.462155+0000","last_fullsized":"2024-03-30T13:11:47.462155+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:11:24.586946+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:11:24.586946+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:34.982227+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":463,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-30T13:11:44.601312+0000","last_change":"2024-03-30T13:11:15.422620+0000","last_active":"2024-03-30T13:11:44.601312+0000","last_peered":"2024-03-30T13:11:44.601312+0000","last_clean":"2024-03-30T13:11:44.601312+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:11:44.601312+0000","last_undegraded":"2024-03-30T13:11:44.601312+0000","last_fullsized":"2024-03-30T13:11:44.601312+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:11:15.422350+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:11:15.422350+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:17.684567+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":353,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-30T13:11:47.462174+0000","last_change":"2024-03-30T13:10:48.606161+0000","last_active":"2024-03-30T13:11:47.462174+0000","last_peered":"2024-03-30T13:11:47.462174+0000","last_clean":"2024-03-30T13:11:47.462174+0000","last_became_active":"2024-03-30T13:09:27.162655+0000","last_became_peered":"2024-03-30T13:09:27.162655+0000","last_unstale":"2024-03-30T13:11:47.462174+0000","last_undegraded":"2024-03-30T13:11:47.462174+0000","last_fullsized":"2024-03-30T13:11:47.462174+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:10:48.605866+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:11:55.059047+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":258,"reported_epoch":439,"state":"active+clean","last_fresh":"2024-03-30T13:11:47.462911+0000","last_change":"2024-03-30T13:11:19.631799+0000","last_active":"2024-03-30T13:11:47.462911+0000","last_peered":"2024-03-30T13:11:47.462911+0000","last_clean":"2024-03-30T13:11:47.462911+0000","last_became_active":"2024-03-30T13:08:39.543465+0000","last_became_peered":"2024-03-30T13:08:39.543465+0000","last_unstale":"2024-03-30T13:11:47.462911+0000","last_undegraded":"2024-03-30T13:11:47.462911+0000","last_fullsized":"2024-03-30T13:11:47.462911+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:19.631559+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:11:19.631559+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:29.111571+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":219177150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":109256704,"data_stored":109256704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834143,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52468,"kb_used_data":20328,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94319372,"statfs":{"total":96636764160,"available":96583036928,"internally_reserved":0,"allocated":20815872,"data_stored":17873452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534771,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57660,"kb_used_data":25072,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94314180,"statfs":{"total":96636764160,"available":96577720320,"internally_reserved":0,"allocated":25673728,"data_stored":22731308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":33350120},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509639,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49468,"kb_used_data":18288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322372,"statfs":{"total":96636764160,"available":96586108928,"internally_reserved":0,"allocated":18726912,"data_stored":15791251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":346,"seq":1486058684457,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44052,"kb_used_data":11912,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94327788,"statfs":{"total":96636764160,"available":96591654912,"internally_reserved":0,"allocated":12197888,"data_stored":9255468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6045,"internal_metadata":32893027},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509639,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50404,"kb_used_data":19224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321436,"statfs":{"total":96636764160,"available":96585150464,"internally_reserved":0,"allocated":19685376,"data_stored":16742956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297034,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52896,"kb_used_data":15252,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94318944,"statfs":{"total":96636764160,"available":96582598656,"internally_reserved":0,"allocated":15618048,"data_stored":12682387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147348,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53260,"kb_used_data":21568,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94318580,"statfs":{"total":96636764160,"available":96582225920,"internally_reserved":0,"allocated":22085632,"data_stored":19149971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14340096,"data_stored":14340096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6852608,"data_stored":6852608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13393920,"data_stored":13393920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16752640,"data_stored":16752640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:11:53.067 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:11:53.067 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:11:53.067 INFO:tasks.thrashosds.thrasher:Reviving osd 4 2024-03-30T13:11:53.067 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-30T13:11:53.067 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-30T13:11:53.070 INFO:tasks.ceph.osd.4:Started 2024-03-30T13:11:53.070 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-30T13:11:53.137 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:11:53.136+0000 7fed4669b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:53.145 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:11:53.144+0000 7fed4669b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:53.145 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:11:53.144+0000 7fed4669b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:53.192 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T13:11:53.192 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T13:11:53.192 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T13:11:53.192 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T13:11:53.192 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T13:11:53.192 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T13:11:53.192 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-03-30T13:11:53.192 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-03-30T13:11:53.192 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-03-30T13:11:53.192 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-03-30T13:11:53.193 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-03-30T13:11:53.193 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T13:11:53.194 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:11:53.195 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-03-30T13:11:53.681 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:11:53.680+0000 7fed4669b7c0 -1 Falling back to public interface 2024-03-30T13:11:57.418 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:11:57.416+0000 7fed4669b7c0 -1 osd.4 435 log_to_monitors true 2024-03-30T13:11:58.185 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:11:58.184+0000 7fed3e470640 -1 osd.4 435 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:11:58.195 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-30T13:11:58.325 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T13:11:58.325 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T13:11:58.325 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T13:11:58.325 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T13:11:58.339 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:11:58.339 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-03-30T13:11:58.339 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 4 2024-03-30T13:11:58.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:58.467+0000 7fd3ad2d2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:58.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:58.479+0000 7fd3ad2d2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:58.665 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:11:58.663+0000 7fbb38f08640 -1 mon.a@0(leader).osd e440 definitely_dead 0 2024-03-30T13:11:59.161 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.4. 2024-03-30T13:11:59.192 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-03-30T13:11:59.192 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:11:59.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:59.307+0000 7f5bc08fb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:11:59.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:11:59.323+0000 7f5bc08fb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:00.186 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:12:00.215 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:12:00.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:00.327+0000 7f720bd08640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:00.343+0000 7f720bd08640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:01.188 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:12:01.214 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:12:01.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:01.323+0000 7f9269a9f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:01.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:01.339+0000 7f9269a9f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:02.208 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:12:02.234 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:12:02.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:02.351+0000 7f2cacd40640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:02.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:02.367+0000 7f2cacd40640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:03.212 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:12:03.240 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:12:03.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:03.359+0000 7f0fadd7e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:03.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:03.375+0000 7f0fadd7e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:04.228 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:12:04.255 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:12:04.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:04.371+0000 7fa437d40640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:04.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:04.387+0000 7fa437d40640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:05.232 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:12:05.258 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:12:05.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:05.375+0000 7f3c79a42640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:05.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:05.391+0000 7f3c79a42640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:06.254 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:12:06.280 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:12:06.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:06.399+0000 7f3f89b8c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:06.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:06.415+0000 7f3f89b8c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:07.266 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:12:07.290 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:12:07.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:07.403+0000 7fb34cf21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:07.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:07.419+0000 7fb34cf21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:08.276 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:12:08.305 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:12:08.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:08.419+0000 7ff37cc7f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:08.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:08.435+0000 7ff37cc7f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:09.289 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:12:09.317 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:12:09.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:09.435+0000 7f9ee1a64640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:09.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:09.455+0000 7f9ee1a64640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:10.297 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:12:10.329 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:12:10.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:10.451+0000 7f7e25d3c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:10.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:10.467+0000 7f7e25d3c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:11.306 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:12:11.335 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:12:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:11.455+0000 7f3bacd46640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:11.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:11.471+0000 7f3bacd46640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:12.317 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:12:12.343 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:12:12.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:12.463+0000 7f21f131b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:12.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:12.479+0000 7f21f131b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:13.322 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:12:13.346 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:12:13.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:13.462+0000 7f1fdad30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:13.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:13.478+0000 7f1fdad30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:14.337 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:12:14.362 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:12:14.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:14.482+0000 7fee2066a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:14.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:14.498+0000 7fee2066a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:15.353 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:12:15.379 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:12:15.379 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:12:15.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:15.494+0000 7f5d690e6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:15.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:15.510+0000 7f5d690e6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:15.687 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:12:15.688 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:12:15.705 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":951,"stamp":"2024-03-30T13:12:15.351946+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":219177150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":407612,"kb_used_data":137672,"kb_used_omap":48,"kb_used_meta":269775,"kb_avail":754567108,"statfs":{"total":773094113280,"available":772676718592,"internally_reserved":0,"allocated":140976128,"data_stored":116642321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":49364,"internal_metadata":276250412},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.046092"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1016,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-30T13:12:10.615099+0000","last_change":"2024-03-30T13:12:00.206765+0000","last_active":"2024-03-30T13:12:10.615099+0000","last_peered":"2024-03-30T13:12:10.615099+0000","last_clean":"2024-03-30T13:12:10.615099+0000","last_became_active":"2024-03-30T13:12:00.205621+0000","last_became_peered":"2024-03-30T13:12:00.205621+0000","last_unstale":"2024-03-30T13:12:10.615099+0000","last_undegraded":"2024-03-30T13:12:10.615099+0000","last_fullsized":"2024-03-30T13:12:10.615099+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:37.447802+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:11:37.447802+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:53.415617+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":976,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-30T13:12:13.328638+0000","last_change":"2024-03-30T13:12:09.454216+0000","last_active":"2024-03-30T13:12:13.328638+0000","last_peered":"2024-03-30T13:12:13.328638+0000","last_clean":"2024-03-30T13:12:13.328638+0000","last_became_active":"2024-03-30T13:12:00.216256+0000","last_became_peered":"2024-03-30T13:12:00.216256+0000","last_unstale":"2024-03-30T13:12:13.328638+0000","last_undegraded":"2024-03-30T13:12:13.328638+0000","last_fullsized":"2024-03-30T13:12:13.328638+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:09.453958+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_clean_scrub_stamp":"2024-03-30T13:12:09.453958+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:33.947990+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":568,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-30T13:12:11.306333+0000","last_change":"2024-03-30T13:12:00.319564+0000","last_active":"2024-03-30T13:12:11.306333+0000","last_peered":"2024-03-30T13:12:11.306333+0000","last_clean":"2024-03-30T13:12:11.306333+0000","last_became_active":"2024-03-30T13:12:00.205139+0000","last_became_peered":"2024-03-30T13:12:00.205139+0000","last_unstale":"2024-03-30T13:12:11.306333+0000","last_undegraded":"2024-03-30T13:12:11.306333+0000","last_fullsized":"2024-03-30T13:12:11.306333+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:12:00.319122+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:12:00.319122+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:20.173841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":522,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-30T13:12:11.306198+0000","last_change":"2024-03-30T13:11:10.457971+0000","last_active":"2024-03-30T13:12:11.306198+0000","last_peered":"2024-03-30T13:12:11.306198+0000","last_clean":"2024-03-30T13:12:11.306198+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:12:11.306198+0000","last_undegraded":"2024-03-30T13:12:11.306198+0000","last_fullsized":"2024-03-30T13:12:11.306198+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:10.457629+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:11:10.457629+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:22.474745+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":345,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-30T13:12:14.058684+0000","last_change":"2024-03-30T13:11:02.527083+0000","last_active":"2024-03-30T13:12:14.058684+0000","last_peered":"2024-03-30T13:12:14.058684+0000","last_clean":"2024-03-30T13:12:14.058684+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:12:14.058684+0000","last_undegraded":"2024-03-30T13:12:14.058684+0000","last_fullsized":"2024-03-30T13:12:14.058684+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:11:02.526699+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:11:02.526699+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:19.924188+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1002,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-30T13:12:11.527737+0000","last_change":"2024-03-30T13:11:12.454138+0000","last_active":"2024-03-30T13:12:11.527737+0000","last_peered":"2024-03-30T13:12:11.527737+0000","last_clean":"2024-03-30T13:12:11.527737+0000","last_became_active":"2024-03-30T13:10:18.181149+0000","last_became_peered":"2024-03-30T13:10:18.181149+0000","last_unstale":"2024-03-30T13:12:11.527737+0000","last_undegraded":"2024-03-30T13:12:11.527737+0000","last_fullsized":"2024-03-30T13:12:11.527737+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:11:12.454045+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_clean_scrub_stamp":"2024-03-30T13:11:12.454045+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:14.977683+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":565,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-30T13:12:11.306311+0000","last_change":"2024-03-30T13:12:04.242615+0000","last_active":"2024-03-30T13:12:11.306311+0000","last_peered":"2024-03-30T13:12:11.306311+0000","last_clean":"2024-03-30T13:12:11.306311+0000","last_became_active":"2024-03-30T13:12:00.202949+0000","last_became_peered":"2024-03-30T13:12:00.202949+0000","last_unstale":"2024-03-30T13:12:11.306311+0000","last_undegraded":"2024-03-30T13:12:11.306311+0000","last_fullsized":"2024-03-30T13:12:11.306311+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:12:04.242418+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:12:04.242418+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:12.083440+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":317,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-30T13:12:14.336927+0000","last_change":"2024-03-30T13:11:54.706399+0000","last_active":"2024-03-30T13:12:14.336927+0000","last_peered":"2024-03-30T13:12:14.336927+0000","last_clean":"2024-03-30T13:12:14.336927+0000","last_became_active":"2024-03-30T13:09:27.546829+0000","last_became_peered":"2024-03-30T13:09:27.546829+0000","last_unstale":"2024-03-30T13:12:14.336927+0000","last_undegraded":"2024-03-30T13:12:14.336927+0000","last_fullsized":"2024-03-30T13:12:14.336927+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:11:54.706131+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:11:54.706131+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:02.275841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1128,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-30T13:12:11.527503+0000","last_change":"2024-03-30T13:11:54.781544+0000","last_active":"2024-03-30T13:12:11.527503+0000","last_peered":"2024-03-30T13:12:11.527503+0000","last_clean":"2024-03-30T13:12:11.527503+0000","last_became_active":"2024-03-30T13:11:09.212712+0000","last_became_peered":"2024-03-30T13:11:09.212712+0000","last_unstale":"2024-03-30T13:12:11.527503+0000","last_undegraded":"2024-03-30T13:12:11.527503+0000","last_fullsized":"2024-03-30T13:12:11.527503+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:54.781288+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:11:54.781288+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:21.599312+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":55832128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":850,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-30T13:12:13.328378+0000","last_change":"2024-03-30T13:12:05.457652+0000","last_active":"2024-03-30T13:12:13.328378+0000","last_peered":"2024-03-30T13:12:13.328378+0000","last_clean":"2024-03-30T13:12:13.328378+0000","last_became_active":"2024-03-30T13:12:00.203605+0000","last_became_peered":"2024-03-30T13:12:00.203605+0000","last_unstale":"2024-03-30T13:12:13.328378+0000","last_undegraded":"2024-03-30T13:12:13.328378+0000","last_fullsized":"2024-03-30T13:12:13.328378+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:05.457409+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:12:05.457409+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:32.879605+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1029,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-30T13:12:10.614915+0000","last_change":"2024-03-30T13:12:00.204472+0000","last_active":"2024-03-30T13:12:10.614915+0000","last_peered":"2024-03-30T13:12:10.614915+0000","last_clean":"2024-03-30T13:12:10.614915+0000","last_became_active":"2024-03-30T13:12:00.202952+0000","last_became_peered":"2024-03-30T13:12:00.202952+0000","last_unstale":"2024-03-30T13:12:10.614915+0000","last_undegraded":"2024-03-30T13:12:10.614915+0000","last_fullsized":"2024-03-30T13:12:10.614915+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:11:20.345991+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:11:20.345991+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:26.201675+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1070,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-30T13:12:10.614948+0000","last_change":"2024-03-30T13:11:13.269830+0000","last_active":"2024-03-30T13:12:10.614948+0000","last_peered":"2024-03-30T13:12:10.614948+0000","last_clean":"2024-03-30T13:12:10.614948+0000","last_became_active":"2024-03-30T13:10:18.180670+0000","last_became_peered":"2024-03-30T13:10:18.180670+0000","last_unstale":"2024-03-30T13:12:10.614948+0000","last_undegraded":"2024-03-30T13:12:10.614948+0000","last_fullsized":"2024-03-30T13:12:10.614948+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:11:13.269449+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:11:13.269449+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:32.598288+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":383,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-30T13:12:14.058685+0000","last_change":"2024-03-30T13:11:24.587378+0000","last_active":"2024-03-30T13:12:14.058685+0000","last_peered":"2024-03-30T13:12:14.058685+0000","last_clean":"2024-03-30T13:12:14.058685+0000","last_became_active":"2024-03-30T13:10:18.183814+0000","last_became_peered":"2024-03-30T13:10:18.183814+0000","last_unstale":"2024-03-30T13:12:14.058685+0000","last_undegraded":"2024-03-30T13:12:14.058685+0000","last_fullsized":"2024-03-30T13:12:14.058685+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:11:24.586946+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:11:24.586946+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:34.982227+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":493,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-30T13:12:11.306221+0000","last_change":"2024-03-30T13:11:15.422620+0000","last_active":"2024-03-30T13:12:11.306221+0000","last_peered":"2024-03-30T13:12:11.306221+0000","last_clean":"2024-03-30T13:12:11.306221+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:12:11.306221+0000","last_undegraded":"2024-03-30T13:12:11.306221+0000","last_fullsized":"2024-03-30T13:12:11.306221+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:11:15.422350+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:11:15.422350+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:17.684567+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":391,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-30T13:12:14.058715+0000","last_change":"2024-03-30T13:11:55.853801+0000","last_active":"2024-03-30T13:12:14.058715+0000","last_peered":"2024-03-30T13:12:14.058715+0000","last_clean":"2024-03-30T13:12:14.058715+0000","last_became_active":"2024-03-30T13:09:27.162655+0000","last_became_peered":"2024-03-30T13:09:27.162655+0000","last_unstale":"2024-03-30T13:12:14.058715+0000","last_undegraded":"2024-03-30T13:12:14.058715+0000","last_fullsized":"2024-03-30T13:12:14.058715+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:11:55.853519+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:11:55.853519+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:05.040598+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":292,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-30T13:12:14.336936+0000","last_change":"2024-03-30T13:11:19.631799+0000","last_active":"2024-03-30T13:12:14.336936+0000","last_peered":"2024-03-30T13:12:14.336936+0000","last_clean":"2024-03-30T13:12:14.336936+0000","last_became_active":"2024-03-30T13:08:39.543465+0000","last_became_peered":"2024-03-30T13:08:39.543465+0000","last_unstale":"2024-03-30T13:12:14.336936+0000","last_undegraded":"2024-03-30T13:12:14.336936+0000","last_fullsized":"2024-03-30T13:12:14.336936+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:19.631559+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:11:19.631559+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:29.111571+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":219177150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834147,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52624,"kb_used_data":20484,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94319216,"statfs":{"total":96636764160,"available":96582877184,"internally_reserved":0,"allocated":20975616,"data_stored":17946642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534776,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61992,"kb_used_data":25240,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94309848,"statfs":{"total":96636764160,"available":96573284352,"internally_reserved":0,"allocated":25845760,"data_stored":22809985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":37609960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":441,"seq":1894080577541,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54164,"kb_used_data":16968,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94317676,"statfs":{"total":96636764160,"available":96581300224,"internally_reserved":0,"allocated":17375232,"data_stored":14325855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9343,"internal_metadata":38067073},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509644,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42936,"kb_used_data":11748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328904,"statfs":{"total":96636764160,"available":96592797696,"internally_reserved":0,"allocated":12029952,"data_stored":8973774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":346,"seq":1486058684462,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43152,"kb_used_data":11012,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94328688,"statfs":{"total":96636764160,"available":96592576512,"internally_reserved":0,"allocated":11276288,"data_stored":8226911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6045,"internal_metadata":32893027},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509644,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50080,"kb_used_data":18900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321760,"statfs":{"total":96636764160,"available":96585482240,"internally_reserved":0,"allocated":19353600,"data_stored":16311024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297039,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53080,"kb_used_data":15432,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94318760,"statfs":{"total":96636764160,"available":96582410240,"internally_reserved":0,"allocated":15802368,"data_stored":12766593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147353,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49584,"kb_used_data":17888,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94322256,"statfs":{"total":96636764160,"available":96585990144,"internally_reserved":0,"allocated":18317312,"data_stored":15281537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:12:15.707 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:12:15.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:15.810+0000 7f9af03a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:15.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:15.822+0000 7f9af03a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:15.999 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:12:15.999 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:12:16.016 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":951,"stamp":"2024-03-30T13:12:15.351946+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":219177150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":407612,"kb_used_data":137672,"kb_used_omap":48,"kb_used_meta":269775,"kb_avail":754567108,"statfs":{"total":773094113280,"available":772676718592,"internally_reserved":0,"allocated":140976128,"data_stored":116642321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":49364,"internal_metadata":276250412},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.046092"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1016,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-30T13:12:10.615099+0000","last_change":"2024-03-30T13:12:00.206765+0000","last_active":"2024-03-30T13:12:10.615099+0000","last_peered":"2024-03-30T13:12:10.615099+0000","last_clean":"2024-03-30T13:12:10.615099+0000","last_became_active":"2024-03-30T13:12:00.205621+0000","last_became_peered":"2024-03-30T13:12:00.205621+0000","last_unstale":"2024-03-30T13:12:10.615099+0000","last_undegraded":"2024-03-30T13:12:10.615099+0000","last_fullsized":"2024-03-30T13:12:10.615099+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:37.447802+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:11:37.447802+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:53.415617+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":976,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-30T13:12:13.328638+0000","last_change":"2024-03-30T13:12:09.454216+0000","last_active":"2024-03-30T13:12:13.328638+0000","last_peered":"2024-03-30T13:12:13.328638+0000","last_clean":"2024-03-30T13:12:13.328638+0000","last_became_active":"2024-03-30T13:12:00.216256+0000","last_became_peered":"2024-03-30T13:12:00.216256+0000","last_unstale":"2024-03-30T13:12:13.328638+0000","last_undegraded":"2024-03-30T13:12:13.328638+0000","last_fullsized":"2024-03-30T13:12:13.328638+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:09.453958+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_clean_scrub_stamp":"2024-03-30T13:12:09.453958+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:33.947990+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":568,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-30T13:12:11.306333+0000","last_change":"2024-03-30T13:12:00.319564+0000","last_active":"2024-03-30T13:12:11.306333+0000","last_peered":"2024-03-30T13:12:11.306333+0000","last_clean":"2024-03-30T13:12:11.306333+0000","last_became_active":"2024-03-30T13:12:00.205139+0000","last_became_peered":"2024-03-30T13:12:00.205139+0000","last_unstale":"2024-03-30T13:12:11.306333+0000","last_undegraded":"2024-03-30T13:12:11.306333+0000","last_fullsized":"2024-03-30T13:12:11.306333+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:12:00.319122+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:12:00.319122+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:20.173841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":522,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-30T13:12:11.306198+0000","last_change":"2024-03-30T13:11:10.457971+0000","last_active":"2024-03-30T13:12:11.306198+0000","last_peered":"2024-03-30T13:12:11.306198+0000","last_clean":"2024-03-30T13:12:11.306198+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:12:11.306198+0000","last_undegraded":"2024-03-30T13:12:11.306198+0000","last_fullsized":"2024-03-30T13:12:11.306198+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:10.457629+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:11:10.457629+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:22.474745+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":345,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-30T13:12:14.058684+0000","last_change":"2024-03-30T13:11:02.527083+0000","last_active":"2024-03-30T13:12:14.058684+0000","last_peered":"2024-03-30T13:12:14.058684+0000","last_clean":"2024-03-30T13:12:14.058684+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:12:14.058684+0000","last_undegraded":"2024-03-30T13:12:14.058684+0000","last_fullsized":"2024-03-30T13:12:14.058684+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:11:02.526699+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:11:02.526699+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:19.924188+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1002,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-30T13:12:11.527737+0000","last_change":"2024-03-30T13:11:12.454138+0000","last_active":"2024-03-30T13:12:11.527737+0000","last_peered":"2024-03-30T13:12:11.527737+0000","last_clean":"2024-03-30T13:12:11.527737+0000","last_became_active":"2024-03-30T13:10:18.181149+0000","last_became_peered":"2024-03-30T13:10:18.181149+0000","last_unstale":"2024-03-30T13:12:11.527737+0000","last_undegraded":"2024-03-30T13:12:11.527737+0000","last_fullsized":"2024-03-30T13:12:11.527737+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:11:12.454045+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_clean_scrub_stamp":"2024-03-30T13:11:12.454045+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:14.977683+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":565,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-30T13:12:11.306311+0000","last_change":"2024-03-30T13:12:04.242615+0000","last_active":"2024-03-30T13:12:11.306311+0000","last_peered":"2024-03-30T13:12:11.306311+0000","last_clean":"2024-03-30T13:12:11.306311+0000","last_became_active":"2024-03-30T13:12:00.202949+0000","last_became_peered":"2024-03-30T13:12:00.202949+0000","last_unstale":"2024-03-30T13:12:11.306311+0000","last_undegraded":"2024-03-30T13:12:11.306311+0000","last_fullsized":"2024-03-30T13:12:11.306311+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:12:04.242418+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:12:04.242418+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:12.083440+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":317,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-30T13:12:14.336927+0000","last_change":"2024-03-30T13:11:54.706399+0000","last_active":"2024-03-30T13:12:14.336927+0000","last_peered":"2024-03-30T13:12:14.336927+0000","last_clean":"2024-03-30T13:12:14.336927+0000","last_became_active":"2024-03-30T13:09:27.546829+0000","last_became_peered":"2024-03-30T13:09:27.546829+0000","last_unstale":"2024-03-30T13:12:14.336927+0000","last_undegraded":"2024-03-30T13:12:14.336927+0000","last_fullsized":"2024-03-30T13:12:14.336927+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:11:54.706131+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:11:54.706131+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:02.275841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1128,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-30T13:12:11.527503+0000","last_change":"2024-03-30T13:11:54.781544+0000","last_active":"2024-03-30T13:12:11.527503+0000","last_peered":"2024-03-30T13:12:11.527503+0000","last_clean":"2024-03-30T13:12:11.527503+0000","last_became_active":"2024-03-30T13:11:09.212712+0000","last_became_peered":"2024-03-30T13:11:09.212712+0000","last_unstale":"2024-03-30T13:12:11.527503+0000","last_undegraded":"2024-03-30T13:12:11.527503+0000","last_fullsized":"2024-03-30T13:12:11.527503+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:54.781288+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:11:54.781288+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:21.599312+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":55832128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":850,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-30T13:12:13.328378+0000","last_change":"2024-03-30T13:12:05.457652+0000","last_active":"2024-03-30T13:12:13.328378+0000","last_peered":"2024-03-30T13:12:13.328378+0000","last_clean":"2024-03-30T13:12:13.328378+0000","last_became_active":"2024-03-30T13:12:00.203605+0000","last_became_peered":"2024-03-30T13:12:00.203605+0000","last_unstale":"2024-03-30T13:12:13.328378+0000","last_undegraded":"2024-03-30T13:12:13.328378+0000","last_fullsized":"2024-03-30T13:12:13.328378+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:05.457409+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:12:05.457409+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:32.879605+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1029,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-30T13:12:10.614915+0000","last_change":"2024-03-30T13:12:00.204472+0000","last_active":"2024-03-30T13:12:10.614915+0000","last_peered":"2024-03-30T13:12:10.614915+0000","last_clean":"2024-03-30T13:12:10.614915+0000","last_became_active":"2024-03-30T13:12:00.202952+0000","last_became_peered":"2024-03-30T13:12:00.202952+0000","last_unstale":"2024-03-30T13:12:10.614915+0000","last_undegraded":"2024-03-30T13:12:10.614915+0000","last_fullsized":"2024-03-30T13:12:10.614915+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:11:20.345991+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:11:20.345991+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:26.201675+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1070,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-30T13:12:10.614948+0000","last_change":"2024-03-30T13:11:13.269830+0000","last_active":"2024-03-30T13:12:10.614948+0000","last_peered":"2024-03-30T13:12:10.614948+0000","last_clean":"2024-03-30T13:12:10.614948+0000","last_became_active":"2024-03-30T13:10:18.180670+0000","last_became_peered":"2024-03-30T13:10:18.180670+0000","last_unstale":"2024-03-30T13:12:10.614948+0000","last_undegraded":"2024-03-30T13:12:10.614948+0000","last_fullsized":"2024-03-30T13:12:10.614948+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:11:13.269449+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:11:13.269449+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:32.598288+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":383,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-30T13:12:14.058685+0000","last_change":"2024-03-30T13:11:24.587378+0000","last_active":"2024-03-30T13:12:14.058685+0000","last_peered":"2024-03-30T13:12:14.058685+0000","last_clean":"2024-03-30T13:12:14.058685+0000","last_became_active":"2024-03-30T13:10:18.183814+0000","last_became_peered":"2024-03-30T13:10:18.183814+0000","last_unstale":"2024-03-30T13:12:14.058685+0000","last_undegraded":"2024-03-30T13:12:14.058685+0000","last_fullsized":"2024-03-30T13:12:14.058685+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:11:24.586946+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:11:24.586946+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:34.982227+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":493,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-30T13:12:11.306221+0000","last_change":"2024-03-30T13:11:15.422620+0000","last_active":"2024-03-30T13:12:11.306221+0000","last_peered":"2024-03-30T13:12:11.306221+0000","last_clean":"2024-03-30T13:12:11.306221+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:12:11.306221+0000","last_undegraded":"2024-03-30T13:12:11.306221+0000","last_fullsized":"2024-03-30T13:12:11.306221+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:11:15.422350+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:11:15.422350+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:17.684567+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":391,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-30T13:12:14.058715+0000","last_change":"2024-03-30T13:11:55.853801+0000","last_active":"2024-03-30T13:12:14.058715+0000","last_peered":"2024-03-30T13:12:14.058715+0000","last_clean":"2024-03-30T13:12:14.058715+0000","last_became_active":"2024-03-30T13:09:27.162655+0000","last_became_peered":"2024-03-30T13:09:27.162655+0000","last_unstale":"2024-03-30T13:12:14.058715+0000","last_undegraded":"2024-03-30T13:12:14.058715+0000","last_fullsized":"2024-03-30T13:12:14.058715+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:11:55.853519+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:11:55.853519+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:05.040598+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":292,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-30T13:12:14.336936+0000","last_change":"2024-03-30T13:11:19.631799+0000","last_active":"2024-03-30T13:12:14.336936+0000","last_peered":"2024-03-30T13:12:14.336936+0000","last_clean":"2024-03-30T13:12:14.336936+0000","last_became_active":"2024-03-30T13:08:39.543465+0000","last_became_peered":"2024-03-30T13:08:39.543465+0000","last_unstale":"2024-03-30T13:12:14.336936+0000","last_undegraded":"2024-03-30T13:12:14.336936+0000","last_fullsized":"2024-03-30T13:12:14.336936+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:19.631559+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:11:19.631559+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:29.111571+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":219177150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834147,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52624,"kb_used_data":20484,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94319216,"statfs":{"total":96636764160,"available":96582877184,"internally_reserved":0,"allocated":20975616,"data_stored":17946642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534776,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61992,"kb_used_data":25240,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94309848,"statfs":{"total":96636764160,"available":96573284352,"internally_reserved":0,"allocated":25845760,"data_stored":22809985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":37609960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":441,"seq":1894080577541,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54164,"kb_used_data":16968,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94317676,"statfs":{"total":96636764160,"available":96581300224,"internally_reserved":0,"allocated":17375232,"data_stored":14325855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9343,"internal_metadata":38067073},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509644,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42936,"kb_used_data":11748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328904,"statfs":{"total":96636764160,"available":96592797696,"internally_reserved":0,"allocated":12029952,"data_stored":8973774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":346,"seq":1486058684462,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43152,"kb_used_data":11012,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94328688,"statfs":{"total":96636764160,"available":96592576512,"internally_reserved":0,"allocated":11276288,"data_stored":8226911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6045,"internal_metadata":32893027},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509644,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50080,"kb_used_data":18900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321760,"statfs":{"total":96636764160,"available":96585482240,"internally_reserved":0,"allocated":19353600,"data_stored":16311024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297039,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53080,"kb_used_data":15432,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94318760,"statfs":{"total":96636764160,"available":96582410240,"internally_reserved":0,"allocated":15802368,"data_stored":12766593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147353,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49584,"kb_used_data":17888,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94322256,"statfs":{"total":96636764160,"available":96585990144,"internally_reserved":0,"allocated":18317312,"data_stored":15281537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:12:16.017 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:12:16.017 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:12:16.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:16.126+0000 7fe6483a5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:16.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:16.142+0000 7fe6483a5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:16.318 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:12:16.318 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":457,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:12:15.341953+0000","last_up_change":"2024-03-30T13:11:59.153181+0000","last_in_change":"2024-03-30T13:11:59.153181+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":24,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":436,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":83,"last_clean_end":336,"up_from":346,"up_thru":411,"down_at":338,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2287832000},{"type":"v1","addr":"172.21.15.67:6818","nonce":2287832000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2287832000},{"type":"v1","addr":"172.21.15.67:6821","nonce":2287832000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2287832000},{"type":"v1","addr":"172.21.15.67:6828","nonce":2287832000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2287832000},{"type":"v1","addr":"172.21.15.67:6825","nonce":2287832000}]},"public_addr":"172.21.15.67:6818/2287832000","cluster_addr":"172.21.15.67:6821/2287832000","heartbeat_back_addr":"172.21.15.67:6828/2287832000","heartbeat_front_addr":"172.21.15.67:6825/2287832000","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":436,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":388,"up_from":393,"up_thru":419,"down_at":389,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6801","nonce":3837194718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6803","nonce":3837194718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6807","nonce":3837194718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6805","nonce":3837194718}]},"public_addr":"172.21.15.67:6801/3837194718","cluster_addr":"172.21.15.67:6803/3837194718","heartbeat_back_addr":"172.21.15.67:6807/3837194718","heartbeat_front_addr":"172.21.15.67:6805/3837194718","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":250,"last_clean_end":435,"up_from":441,"up_thru":441,"down_at":436,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3152653542},{"type":"v1","addr":"172.21.15.82:6803","nonce":3152653542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":3152653542},{"type":"v1","addr":"172.21.15.82:6807","nonce":3152653542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":3152653542},{"type":"v1","addr":"172.21.15.82:6815","nonce":3152653542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":3152653542},{"type":"v1","addr":"172.21.15.82:6811","nonce":3152653542}]},"public_addr":"172.21.15.82:6803/3152653542","cluster_addr":"172.21.15.82:6807/3152653542","heartbeat_back_addr":"172.21.15.82:6815/3152653542","heartbeat_front_addr":"172.21.15.82:6811/3152653542","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":274,"last_clean_end":314,"up_from":320,"up_thru":411,"down_at":315,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":925466951},{"type":"v1","addr":"172.21.15.82:6817","nonce":925466951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":925466951},{"type":"v1","addr":"172.21.15.82:6819","nonce":925466951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":925466951},{"type":"v1","addr":"172.21.15.82:6823","nonce":925466951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":925466951},{"type":"v1","addr":"172.21.15.82:6821","nonce":925466951}]},"public_addr":"172.21.15.82:6817/925466951","cluster_addr":"172.21.15.82:6819/925466951","heartbeat_back_addr":"172.21.15.82:6823/925466951","heartbeat_front_addr":"172.21.15.82:6821/925466951","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":227,"last_clean_end":363,"up_from":372,"up_thru":441,"down_at":364,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6801","nonce":189641108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6806","nonce":189641108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6814","nonce":189641108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6810","nonce":189641108}]},"public_addr":"172.21.15.82:6801/189641108","cluster_addr":"172.21.15.82:6806/189641108","heartbeat_back_addr":"172.21.15.82:6814/189641108","heartbeat_front_addr":"172.21.15.82:6810/189641108","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":298,"last_clean_end":409,"up_from":419,"up_thru":441,"down_at":410,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6825","nonce":2166599911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6827","nonce":2166599911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6831","nonce":2166599911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6829","nonce":2166599911}]},"public_addr":"172.21.15.82:6825/2166599911","cluster_addr":"172.21.15.82:6827/2166599911","heartbeat_back_addr":"172.21.15.82:6831/2166599911","heartbeat_front_addr":"172.21.15.82:6829/2166599911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:12:04.096476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:12:03.772581+0000","dead_epoch":338},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:12:09.071073+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:11:43.966354+0000","dead_epoch":389},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:11:58.441017+0000","dead_epoch":436},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:12:08.962115+0000","dead_epoch":315},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:11:46.639909+0000","dead_epoch":364},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:12:04.878998+0000","dead_epoch":410}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:12:16.336 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:12:16.336 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:12:16.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:16.442+0000 7f1a95329640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:16.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:16.458+0000 7f1a95329640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:16.626 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:12:16.643 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:12:16.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:16.754+0000 7fd74c6f4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:16.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:16.770+0000 7fd74c6f4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:16.951 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:12:16.951 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:12:16.970 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:12:16.970 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:12:16.970 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:12:16.970 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:12:16.970 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 5, 1, 6, 3, 7, 4] 2024-03-30T13:12:16.970 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:12:16.970 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:12:17.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:17.078+0000 7f5fc0af2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:17.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:17.090+0000 7f5fc0af2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:17.267 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:12:17.268 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:12:17.285 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":952,"stamp":"2024-03-30T13:12:16.071077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":219177150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":407612,"kb_used_data":137672,"kb_used_omap":48,"kb_used_meta":269775,"kb_avail":754567108,"statfs":{"total":773094113280,"available":772676718592,"internally_reserved":0,"allocated":140976128,"data_stored":116642321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":49364,"internal_metadata":276250412},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001014"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1016,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-30T13:12:10.615099+0000","last_change":"2024-03-30T13:12:00.206765+0000","last_active":"2024-03-30T13:12:10.615099+0000","last_peered":"2024-03-30T13:12:10.615099+0000","last_clean":"2024-03-30T13:12:10.615099+0000","last_became_active":"2024-03-30T13:12:00.205621+0000","last_became_peered":"2024-03-30T13:12:00.205621+0000","last_unstale":"2024-03-30T13:12:10.615099+0000","last_undegraded":"2024-03-30T13:12:10.615099+0000","last_fullsized":"2024-03-30T13:12:10.615099+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:37.447802+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:11:37.447802+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:53.415617+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":976,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-30T13:12:13.328638+0000","last_change":"2024-03-30T13:12:09.454216+0000","last_active":"2024-03-30T13:12:13.328638+0000","last_peered":"2024-03-30T13:12:13.328638+0000","last_clean":"2024-03-30T13:12:13.328638+0000","last_became_active":"2024-03-30T13:12:00.216256+0000","last_became_peered":"2024-03-30T13:12:00.216256+0000","last_unstale":"2024-03-30T13:12:13.328638+0000","last_undegraded":"2024-03-30T13:12:13.328638+0000","last_fullsized":"2024-03-30T13:12:13.328638+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:09.453958+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_clean_scrub_stamp":"2024-03-30T13:12:09.453958+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:33.947990+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":568,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-30T13:12:11.306333+0000","last_change":"2024-03-30T13:12:00.319564+0000","last_active":"2024-03-30T13:12:11.306333+0000","last_peered":"2024-03-30T13:12:11.306333+0000","last_clean":"2024-03-30T13:12:11.306333+0000","last_became_active":"2024-03-30T13:12:00.205139+0000","last_became_peered":"2024-03-30T13:12:00.205139+0000","last_unstale":"2024-03-30T13:12:11.306333+0000","last_undegraded":"2024-03-30T13:12:11.306333+0000","last_fullsized":"2024-03-30T13:12:11.306333+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:12:00.319122+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:12:00.319122+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:20.173841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":522,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-30T13:12:11.306198+0000","last_change":"2024-03-30T13:11:10.457971+0000","last_active":"2024-03-30T13:12:11.306198+0000","last_peered":"2024-03-30T13:12:11.306198+0000","last_clean":"2024-03-30T13:12:11.306198+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:12:11.306198+0000","last_undegraded":"2024-03-30T13:12:11.306198+0000","last_fullsized":"2024-03-30T13:12:11.306198+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:10.457629+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:11:10.457629+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:22.474745+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":345,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-30T13:12:14.058684+0000","last_change":"2024-03-30T13:11:02.527083+0000","last_active":"2024-03-30T13:12:14.058684+0000","last_peered":"2024-03-30T13:12:14.058684+0000","last_clean":"2024-03-30T13:12:14.058684+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:12:14.058684+0000","last_undegraded":"2024-03-30T13:12:14.058684+0000","last_fullsized":"2024-03-30T13:12:14.058684+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:11:02.526699+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:11:02.526699+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:19.924188+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1002,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-30T13:12:11.527737+0000","last_change":"2024-03-30T13:11:12.454138+0000","last_active":"2024-03-30T13:12:11.527737+0000","last_peered":"2024-03-30T13:12:11.527737+0000","last_clean":"2024-03-30T13:12:11.527737+0000","last_became_active":"2024-03-30T13:10:18.181149+0000","last_became_peered":"2024-03-30T13:10:18.181149+0000","last_unstale":"2024-03-30T13:12:11.527737+0000","last_undegraded":"2024-03-30T13:12:11.527737+0000","last_fullsized":"2024-03-30T13:12:11.527737+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:11:12.454045+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:40.613260+0000","last_clean_scrub_stamp":"2024-03-30T13:11:12.454045+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:14.977683+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":565,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-30T13:12:11.306311+0000","last_change":"2024-03-30T13:12:04.242615+0000","last_active":"2024-03-30T13:12:11.306311+0000","last_peered":"2024-03-30T13:12:11.306311+0000","last_clean":"2024-03-30T13:12:11.306311+0000","last_became_active":"2024-03-30T13:12:00.202949+0000","last_became_peered":"2024-03-30T13:12:00.202949+0000","last_unstale":"2024-03-30T13:12:11.306311+0000","last_undegraded":"2024-03-30T13:12:11.306311+0000","last_fullsized":"2024-03-30T13:12:11.306311+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:12:04.242418+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:12:04.242418+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:12.083440+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":317,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-30T13:12:14.336927+0000","last_change":"2024-03-30T13:11:54.706399+0000","last_active":"2024-03-30T13:12:14.336927+0000","last_peered":"2024-03-30T13:12:14.336927+0000","last_clean":"2024-03-30T13:12:14.336927+0000","last_became_active":"2024-03-30T13:09:27.546829+0000","last_became_peered":"2024-03-30T13:09:27.546829+0000","last_unstale":"2024-03-30T13:12:14.336927+0000","last_undegraded":"2024-03-30T13:12:14.336927+0000","last_fullsized":"2024-03-30T13:12:14.336927+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:11:54.706131+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:11:54.706131+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:02.275841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1128,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-30T13:12:11.527503+0000","last_change":"2024-03-30T13:11:54.781544+0000","last_active":"2024-03-30T13:12:11.527503+0000","last_peered":"2024-03-30T13:12:11.527503+0000","last_clean":"2024-03-30T13:12:11.527503+0000","last_became_active":"2024-03-30T13:11:09.212712+0000","last_became_peered":"2024-03-30T13:11:09.212712+0000","last_unstale":"2024-03-30T13:12:11.527503+0000","last_undegraded":"2024-03-30T13:12:11.527503+0000","last_fullsized":"2024-03-30T13:12:11.527503+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:54.781288+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:11:54.781288+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:21.599312+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":55832128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":850,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-30T13:12:13.328378+0000","last_change":"2024-03-30T13:12:05.457652+0000","last_active":"2024-03-30T13:12:13.328378+0000","last_peered":"2024-03-30T13:12:13.328378+0000","last_clean":"2024-03-30T13:12:13.328378+0000","last_became_active":"2024-03-30T13:12:00.203605+0000","last_became_peered":"2024-03-30T13:12:00.203605+0000","last_unstale":"2024-03-30T13:12:13.328378+0000","last_undegraded":"2024-03-30T13:12:13.328378+0000","last_fullsized":"2024-03-30T13:12:13.328378+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:05.457409+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:12:05.457409+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:32.879605+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1029,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-30T13:12:10.614915+0000","last_change":"2024-03-30T13:12:00.204472+0000","last_active":"2024-03-30T13:12:10.614915+0000","last_peered":"2024-03-30T13:12:10.614915+0000","last_clean":"2024-03-30T13:12:10.614915+0000","last_became_active":"2024-03-30T13:12:00.202952+0000","last_became_peered":"2024-03-30T13:12:00.202952+0000","last_unstale":"2024-03-30T13:12:10.614915+0000","last_undegraded":"2024-03-30T13:12:10.614915+0000","last_fullsized":"2024-03-30T13:12:10.614915+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:11:20.345991+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:11:20.345991+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:26.201675+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1070,"reported_epoch":452,"state":"active+clean","last_fresh":"2024-03-30T13:12:10.614948+0000","last_change":"2024-03-30T13:11:13.269830+0000","last_active":"2024-03-30T13:12:10.614948+0000","last_peered":"2024-03-30T13:12:10.614948+0000","last_clean":"2024-03-30T13:12:10.614948+0000","last_became_active":"2024-03-30T13:10:18.180670+0000","last_became_peered":"2024-03-30T13:10:18.180670+0000","last_unstale":"2024-03-30T13:12:10.614948+0000","last_undegraded":"2024-03-30T13:12:10.614948+0000","last_fullsized":"2024-03-30T13:12:10.614948+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:11:13.269449+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:11:13.269449+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:32.598288+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":383,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-30T13:12:14.058685+0000","last_change":"2024-03-30T13:11:24.587378+0000","last_active":"2024-03-30T13:12:14.058685+0000","last_peered":"2024-03-30T13:12:14.058685+0000","last_clean":"2024-03-30T13:12:14.058685+0000","last_became_active":"2024-03-30T13:10:18.183814+0000","last_became_peered":"2024-03-30T13:10:18.183814+0000","last_unstale":"2024-03-30T13:12:14.058685+0000","last_undegraded":"2024-03-30T13:12:14.058685+0000","last_fullsized":"2024-03-30T13:12:14.058685+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:11:24.586946+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:11:24.586946+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:34.982227+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":493,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-03-30T13:12:11.306221+0000","last_change":"2024-03-30T13:11:15.422620+0000","last_active":"2024-03-30T13:12:11.306221+0000","last_peered":"2024-03-30T13:12:11.306221+0000","last_clean":"2024-03-30T13:12:11.306221+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:12:11.306221+0000","last_undegraded":"2024-03-30T13:12:11.306221+0000","last_fullsized":"2024-03-30T13:12:11.306221+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:11:15.422350+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:11:15.422350+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:17.684567+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":391,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-30T13:12:14.058715+0000","last_change":"2024-03-30T13:11:55.853801+0000","last_active":"2024-03-30T13:12:14.058715+0000","last_peered":"2024-03-30T13:12:14.058715+0000","last_clean":"2024-03-30T13:12:14.058715+0000","last_became_active":"2024-03-30T13:09:27.162655+0000","last_became_peered":"2024-03-30T13:09:27.162655+0000","last_unstale":"2024-03-30T13:12:14.058715+0000","last_undegraded":"2024-03-30T13:12:14.058715+0000","last_fullsized":"2024-03-30T13:12:14.058715+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:11:55.853519+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:11:55.853519+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:05.040598+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":292,"reported_epoch":456,"state":"active+clean","last_fresh":"2024-03-30T13:12:14.336936+0000","last_change":"2024-03-30T13:11:19.631799+0000","last_active":"2024-03-30T13:12:14.336936+0000","last_peered":"2024-03-30T13:12:14.336936+0000","last_clean":"2024-03-30T13:12:14.336936+0000","last_became_active":"2024-03-30T13:08:39.543465+0000","last_became_peered":"2024-03-30T13:08:39.543465+0000","last_unstale":"2024-03-30T13:12:14.336936+0000","last_undegraded":"2024-03-30T13:12:14.336936+0000","last_fullsized":"2024-03-30T13:12:14.336936+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:19.631559+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:11:19.631559+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:29.111571+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":219177150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834147,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52624,"kb_used_data":20484,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94319216,"statfs":{"total":96636764160,"available":96582877184,"internally_reserved":0,"allocated":20975616,"data_stored":17946642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534776,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61992,"kb_used_data":25240,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94309848,"statfs":{"total":96636764160,"available":96573284352,"internally_reserved":0,"allocated":25845760,"data_stored":22809985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":37609960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":441,"seq":1894080577541,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54164,"kb_used_data":16968,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94317676,"statfs":{"total":96636764160,"available":96581300224,"internally_reserved":0,"allocated":17375232,"data_stored":14325855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9343,"internal_metadata":38067073},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509644,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42936,"kb_used_data":11748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328904,"statfs":{"total":96636764160,"available":96592797696,"internally_reserved":0,"allocated":12029952,"data_stored":8973774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":346,"seq":1486058684462,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43152,"kb_used_data":11012,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94328688,"statfs":{"total":96636764160,"available":96592576512,"internally_reserved":0,"allocated":11276288,"data_stored":8226911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6045,"internal_metadata":32893027},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509644,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50080,"kb_used_data":18900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321760,"statfs":{"total":96636764160,"available":96585482240,"internally_reserved":0,"allocated":19353600,"data_stored":16311024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297039,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53080,"kb_used_data":15432,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94318760,"statfs":{"total":96636764160,"available":96582410240,"internally_reserved":0,"allocated":15802368,"data_stored":12766593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147353,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49584,"kb_used_data":17888,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94322256,"statfs":{"total":96636764160,"available":96585990144,"internally_reserved":0,"allocated":18317312,"data_stored":15281537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:12:17.286 INFO:tasks.thrashosds.thrasher:Choosing PG 1.3 with acting set [1, 6, 5] 2024-03-30T13:12:17.286 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 2, 5, 1, 6, 3, 7, 4] 2024-03-30T13:12:17.287 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-30T13:12:17.287 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:12:17.380 INFO:tasks.ceph.osd.1:Stopped 2024-03-30T13:12:17.380 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 2, 5, 1, 6, 3, 7, 4] 2024-03-30T13:12:17.380 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 1 2024-03-30T13:12:17.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:17.482+0000 7f1a4e8e9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:17.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:17.498+0000 7f1a4e8e9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:17.683 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:12:17.678+0000 7fbb38f08640 -1 mon.a@0(leader).osd e457 definitely_dead 0 2024-03-30T13:12:18.374 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.1. 2024-03-30T13:12:18.396 INFO:tasks.thrashosds.thrasher:dead_osds=[1], live_osds=[0, 2, 5, 6, 3, 7, 4] 2024-03-30T13:12:18.397 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:12:18.397 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:12:18.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:18.510+0000 7f3fb3842640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:18.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:18.526+0000 7f3fb3842640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:18.701 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:12:18.702 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:12:18.719 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":954,"stamp":"2024-03-30T13:12:18.378325+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":219177150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":45,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":364584,"kb_used_data":126792,"kb_used_omap":42,"kb_used_meta":237653,"kb_avail":660238296,"statfs":{"total":676457349120,"available":676084015104,"internally_reserved":0,"allocated":129835008,"data_stored":108475767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43319,"internal_metadata":243357385},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.052388"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1024,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-30T13:12:15.810219+0000","last_change":"2024-03-30T13:12:00.206765+0000","last_active":"2024-03-30T13:12:15.810219+0000","last_peered":"2024-03-30T13:12:15.810219+0000","last_clean":"2024-03-30T13:12:15.810219+0000","last_became_active":"2024-03-30T13:12:00.205621+0000","last_became_peered":"2024-03-30T13:12:00.205621+0000","last_unstale":"2024-03-30T13:12:15.810219+0000","last_undegraded":"2024-03-30T13:12:15.810219+0000","last_fullsized":"2024-03-30T13:12:15.810219+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:37.447802+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:11:37.447802+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:53.415617+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":976,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-30T13:12:13.328638+0000","last_change":"2024-03-30T13:12:09.454216+0000","last_active":"2024-03-30T13:12:13.328638+0000","last_peered":"2024-03-30T13:12:13.328638+0000","last_clean":"2024-03-30T13:12:13.328638+0000","last_became_active":"2024-03-30T13:12:00.216256+0000","last_became_peered":"2024-03-30T13:12:00.216256+0000","last_unstale":"2024-03-30T13:12:13.328638+0000","last_undegraded":"2024-03-30T13:12:13.328638+0000","last_fullsized":"2024-03-30T13:12:13.328638+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:09.453958+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_clean_scrub_stamp":"2024-03-30T13:12:09.453958+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:33.947990+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":574,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-30T13:12:15.915719+0000","last_change":"2024-03-30T13:12:00.319564+0000","last_active":"2024-03-30T13:12:15.915719+0000","last_peered":"2024-03-30T13:12:15.915719+0000","last_clean":"2024-03-30T13:12:15.915719+0000","last_became_active":"2024-03-30T13:12:00.205139+0000","last_became_peered":"2024-03-30T13:12:00.205139+0000","last_unstale":"2024-03-30T13:12:15.915719+0000","last_undegraded":"2024-03-30T13:12:15.915719+0000","last_fullsized":"2024-03-30T13:12:15.915719+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:12:00.319122+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:12:00.319122+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:20.173841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":528,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-30T13:12:15.915679+0000","last_change":"2024-03-30T13:11:10.457971+0000","last_active":"2024-03-30T13:12:15.915679+0000","last_peered":"2024-03-30T13:12:15.915679+0000","last_clean":"2024-03-30T13:12:15.915679+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:12:15.915679+0000","last_undegraded":"2024-03-30T13:12:15.915679+0000","last_fullsized":"2024-03-30T13:12:15.915679+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:10.457629+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:11:10.457629+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:22.474745+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":345,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-30T13:12:14.058684+0000","last_change":"2024-03-30T13:11:02.527083+0000","last_active":"2024-03-30T13:12:14.058684+0000","last_peered":"2024-03-30T13:12:14.058684+0000","last_clean":"2024-03-30T13:12:14.058684+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:12:14.058684+0000","last_undegraded":"2024-03-30T13:12:14.058684+0000","last_fullsized":"2024-03-30T13:12:14.058684+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:11:02.526699+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:11:02.526699+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:19.924188+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1016,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-30T13:12:15.741501+0000","last_change":"2024-03-30T13:12:15.741547+0000","last_active":"2024-03-30T13:12:15.741501+0000","last_peered":"2024-03-30T13:12:15.741501+0000","last_clean":"2024-03-30T13:12:15.741501+0000","last_became_active":"2024-03-30T13:10:18.181149+0000","last_became_peered":"2024-03-30T13:10:18.181149+0000","last_unstale":"2024-03-30T13:12:15.741501+0000","last_undegraded":"2024-03-30T13:12:15.741501+0000","last_fullsized":"2024-03-30T13:12:15.741501+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:12:15.741379+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:36.529709+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":571,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-30T13:12:15.915681+0000","last_change":"2024-03-30T13:12:04.242615+0000","last_active":"2024-03-30T13:12:15.915681+0000","last_peered":"2024-03-30T13:12:15.915681+0000","last_clean":"2024-03-30T13:12:15.915681+0000","last_became_active":"2024-03-30T13:12:00.202949+0000","last_became_peered":"2024-03-30T13:12:00.202949+0000","last_unstale":"2024-03-30T13:12:15.915681+0000","last_undegraded":"2024-03-30T13:12:15.915681+0000","last_fullsized":"2024-03-30T13:12:15.915681+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:12:04.242418+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:12:04.242418+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:12.083440+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":317,"reported_epoch":456,"state":"stale+active+clean","last_fresh":"2024-03-30T13:12:14.336927+0000","last_change":"2024-03-30T13:11:54.706399+0000","last_active":"2024-03-30T13:12:14.336927+0000","last_peered":"2024-03-30T13:12:14.336927+0000","last_clean":"2024-03-30T13:12:14.336927+0000","last_became_active":"2024-03-30T13:09:27.546829+0000","last_became_peered":"2024-03-30T13:09:27.546829+0000","last_unstale":"2024-03-30T13:12:18.378346+0000","last_undegraded":"2024-03-30T13:12:14.336927+0000","last_fullsized":"2024-03-30T13:12:14.336927+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:11:54.706131+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:11:54.706131+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:02.275841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1134,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-30T13:12:15.726314+0000","last_change":"2024-03-30T13:11:54.781544+0000","last_active":"2024-03-30T13:12:15.726314+0000","last_peered":"2024-03-30T13:12:15.726314+0000","last_clean":"2024-03-30T13:12:15.726314+0000","last_became_active":"2024-03-30T13:11:09.212712+0000","last_became_peered":"2024-03-30T13:11:09.212712+0000","last_unstale":"2024-03-30T13:12:15.726314+0000","last_undegraded":"2024-03-30T13:12:15.726314+0000","last_fullsized":"2024-03-30T13:12:15.726314+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:54.781288+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:11:54.781288+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:21.599312+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":55832128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":850,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-30T13:12:13.328378+0000","last_change":"2024-03-30T13:12:05.457652+0000","last_active":"2024-03-30T13:12:13.328378+0000","last_peered":"2024-03-30T13:12:13.328378+0000","last_clean":"2024-03-30T13:12:13.328378+0000","last_became_active":"2024-03-30T13:12:00.203605+0000","last_became_peered":"2024-03-30T13:12:00.203605+0000","last_unstale":"2024-03-30T13:12:13.328378+0000","last_undegraded":"2024-03-30T13:12:13.328378+0000","last_fullsized":"2024-03-30T13:12:13.328378+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:05.457409+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:12:05.457409+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:32.879605+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1037,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-30T13:12:15.810172+0000","last_change":"2024-03-30T13:12:00.204472+0000","last_active":"2024-03-30T13:12:15.810172+0000","last_peered":"2024-03-30T13:12:15.810172+0000","last_clean":"2024-03-30T13:12:15.810172+0000","last_became_active":"2024-03-30T13:12:00.202952+0000","last_became_peered":"2024-03-30T13:12:00.202952+0000","last_unstale":"2024-03-30T13:12:15.810172+0000","last_undegraded":"2024-03-30T13:12:15.810172+0000","last_fullsized":"2024-03-30T13:12:15.810172+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:11:20.345991+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:07:48.693155+0000","last_clean_scrub_stamp":"2024-03-30T13:11:20.345991+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:26.201675+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1078,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-30T13:12:15.810157+0000","last_change":"2024-03-30T13:11:13.269830+0000","last_active":"2024-03-30T13:12:15.810157+0000","last_peered":"2024-03-30T13:12:15.810157+0000","last_clean":"2024-03-30T13:12:15.810157+0000","last_became_active":"2024-03-30T13:10:18.180670+0000","last_became_peered":"2024-03-30T13:10:18.180670+0000","last_unstale":"2024-03-30T13:12:15.810157+0000","last_undegraded":"2024-03-30T13:12:15.810157+0000","last_fullsized":"2024-03-30T13:12:15.810157+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:11:13.269449+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:11:13.269449+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:32.598288+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":383,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-30T13:12:14.058685+0000","last_change":"2024-03-30T13:11:24.587378+0000","last_active":"2024-03-30T13:12:14.058685+0000","last_peered":"2024-03-30T13:12:14.058685+0000","last_clean":"2024-03-30T13:12:14.058685+0000","last_became_active":"2024-03-30T13:10:18.183814+0000","last_became_peered":"2024-03-30T13:10:18.183814+0000","last_unstale":"2024-03-30T13:12:14.058685+0000","last_undegraded":"2024-03-30T13:12:14.058685+0000","last_fullsized":"2024-03-30T13:12:14.058685+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:11:24.586946+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:11:24.586946+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:34.982227+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":507,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-30T13:12:18.090851+0000","last_change":"2024-03-30T13:12:18.090983+0000","last_active":"2024-03-30T13:12:18.090851+0000","last_peered":"2024-03-30T13:12:18.090851+0000","last_clean":"2024-03-30T13:12:18.090851+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:12:18.090851+0000","last_undegraded":"2024-03-30T13:12:18.090851+0000","last_fullsized":"2024-03-30T13:12:18.090851+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:18.090642+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:12:18.090642+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:27.145345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":391,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-30T13:12:14.058715+0000","last_change":"2024-03-30T13:11:55.853801+0000","last_active":"2024-03-30T13:12:14.058715+0000","last_peered":"2024-03-30T13:12:14.058715+0000","last_clean":"2024-03-30T13:12:14.058715+0000","last_became_active":"2024-03-30T13:09:27.162655+0000","last_became_peered":"2024-03-30T13:09:27.162655+0000","last_unstale":"2024-03-30T13:12:14.058715+0000","last_undegraded":"2024-03-30T13:12:14.058715+0000","last_fullsized":"2024-03-30T13:12:14.058715+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:11:55.853519+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:11:55.853519+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:05.040598+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":292,"reported_epoch":456,"state":"stale+active+clean","last_fresh":"2024-03-30T13:12:14.336936+0000","last_change":"2024-03-30T13:11:19.631799+0000","last_active":"2024-03-30T13:12:14.336936+0000","last_peered":"2024-03-30T13:12:14.336936+0000","last_clean":"2024-03-30T13:12:14.336936+0000","last_became_active":"2024-03-30T13:08:39.543465+0000","last_became_peered":"2024-03-30T13:08:39.543465+0000","last_unstale":"2024-03-30T13:12:18.378349+0000","last_undegraded":"2024-03-30T13:12:14.336936+0000","last_fullsized":"2024-03-30T13:12:14.336936+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:19.631559+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:11:19.631559+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:29.111571+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":89,"num_bytes_recovered":219177150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834148,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52660,"kb_used_data":20520,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94319180,"statfs":{"total":96636764160,"available":96582840320,"internally_reserved":0,"allocated":21012480,"data_stored":17963103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534776,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61992,"kb_used_data":25240,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94309848,"statfs":{"total":96636764160,"available":96573284352,"internally_reserved":0,"allocated":25845760,"data_stored":22809985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":37609960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":441,"seq":1894080577541,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54164,"kb_used_data":16968,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94317676,"statfs":{"total":96636764160,"available":96581300224,"internally_reserved":0,"allocated":17375232,"data_stored":14325855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9343,"internal_metadata":38067073},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509644,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42936,"kb_used_data":11748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328904,"statfs":{"total":96636764160,"available":96592797696,"internally_reserved":0,"allocated":12029952,"data_stored":8973774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509644,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50080,"kb_used_data":18900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321760,"statfs":{"total":96636764160,"available":96585482240,"internally_reserved":0,"allocated":19353600,"data_stored":16311024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297040,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53124,"kb_used_data":15480,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94318716,"statfs":{"total":96636764160,"available":96582365184,"internally_reserved":0,"allocated":15851520,"data_stored":12788541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147354,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49628,"kb_used_data":17936,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94322212,"statfs":{"total":96636764160,"available":96585945088,"internally_reserved":0,"allocated":18366464,"data_stored":15303485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:12:18.720 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:12:43.721 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:12:43.722 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:12:43.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:43.850+0000 7fcff2eee640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:43.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:43.862+0000 7fcff2eee640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:44.033 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:12:44.033 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:12:44.051 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":967,"stamp":"2024-03-30T13:12:42.078859+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":232219217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":370468,"kb_used_data":132704,"kb_used_omap":42,"kb_used_meta":237653,"kb_avail":660232412,"statfs":{"total":676457349120,"available":676077989888,"internally_reserved":0,"allocated":135888896,"data_stored":114353788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43319,"internal_metadata":243357385},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003716"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1028,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-30T13:12:19.383079+0000","last_change":"2024-03-30T13:12:00.206765+0000","last_active":"2024-03-30T13:12:19.383079+0000","last_peered":"2024-03-30T13:12:19.383079+0000","last_clean":"2024-03-30T13:12:19.383079+0000","last_became_active":"2024-03-30T13:12:00.205621+0000","last_became_peered":"2024-03-30T13:12:00.205621+0000","last_unstale":"2024-03-30T13:12:19.383079+0000","last_undegraded":"2024-03-30T13:12:19.383079+0000","last_fullsized":"2024-03-30T13:12:19.383079+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:37.447802+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:11:37.447802+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:53.415617+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":984,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-30T13:12:19.386839+0000","last_change":"2024-03-30T13:12:09.454216+0000","last_active":"2024-03-30T13:12:19.386839+0000","last_peered":"2024-03-30T13:12:19.386839+0000","last_clean":"2024-03-30T13:12:19.386839+0000","last_became_active":"2024-03-30T13:12:00.216256+0000","last_became_peered":"2024-03-30T13:12:00.216256+0000","last_unstale":"2024-03-30T13:12:19.386839+0000","last_undegraded":"2024-03-30T13:12:19.386839+0000","last_fullsized":"2024-03-30T13:12:19.386839+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:09.453958+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_clean_scrub_stamp":"2024-03-30T13:12:09.453958+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:33.947990+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":592,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-30T13:12:19.467594+0000","last_change":"2024-03-30T13:12:19.467594+0000","last_active":"2024-03-30T13:12:19.467594+0000","last_peered":"2024-03-30T13:12:19.467594+0000","last_clean":"2024-03-30T13:12:19.467594+0000","last_became_active":"2024-03-30T13:12:19.391352+0000","last_became_peered":"2024-03-30T13:12:19.391352+0000","last_unstale":"2024-03-30T13:12:19.467594+0000","last_undegraded":"2024-03-30T13:12:19.467594+0000","last_fullsized":"2024-03-30T13:12:19.467594+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:12:00.319122+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:12:00.319122+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:22.156573+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":540,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-30T13:12:23.100861+0000","last_change":"2024-03-30T13:12:23.100937+0000","last_active":"2024-03-30T13:12:23.100861+0000","last_peered":"2024-03-30T13:12:23.100861+0000","last_clean":"2024-03-30T13:12:23.100861+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:12:23.100861+0000","last_undegraded":"2024-03-30T13:12:23.100861+0000","last_fullsized":"2024-03-30T13:12:23.100861+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:12:23.100675+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:12:23.100675+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:44.577727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":361,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-30T13:12:20.783826+0000","last_change":"2024-03-30T13:12:20.783890+0000","last_active":"2024-03-30T13:12:20.783826+0000","last_peered":"2024-03-30T13:12:20.783826+0000","last_clean":"2024-03-30T13:12:20.783826+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:12:20.783826+0000","last_undegraded":"2024-03-30T13:12:20.783826+0000","last_fullsized":"2024-03-30T13:12:20.783826+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:20.783652+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:12:20.783652+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:32.641222+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1021,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-30T13:12:19.797497+0000","last_change":"2024-03-30T13:12:15.741547+0000","last_active":"2024-03-30T13:12:19.797497+0000","last_peered":"2024-03-30T13:12:19.797497+0000","last_clean":"2024-03-30T13:12:19.797497+0000","last_became_active":"2024-03-30T13:10:18.181149+0000","last_became_peered":"2024-03-30T13:10:18.181149+0000","last_unstale":"2024-03-30T13:12:19.797497+0000","last_undegraded":"2024-03-30T13:12:19.797497+0000","last_fullsized":"2024-03-30T13:12:19.797497+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:12:15.741379+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:36.529709+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":575,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-30T13:12:19.384014+0000","last_change":"2024-03-30T13:12:04.242615+0000","last_active":"2024-03-30T13:12:19.384014+0000","last_peered":"2024-03-30T13:12:19.384014+0000","last_clean":"2024-03-30T13:12:19.384014+0000","last_became_active":"2024-03-30T13:12:00.202949+0000","last_became_peered":"2024-03-30T13:12:00.202949+0000","last_unstale":"2024-03-30T13:12:19.384014+0000","last_undegraded":"2024-03-30T13:12:19.384014+0000","last_fullsized":"2024-03-30T13:12:19.384014+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:12:04.242418+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:12:04.242418+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:12.083440+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":19,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-30T13:12:19.468098+0000","last_change":"2024-03-30T13:12:19.468098+0000","last_active":"2024-03-30T13:12:19.468098+0000","last_peered":"2024-03-30T13:12:19.468098+0000","last_clean":"2024-03-30T13:12:19.468098+0000","last_became_active":"2024-03-30T13:12:19.393418+0000","last_became_peered":"2024-03-30T13:12:19.393418+0000","last_unstale":"2024-03-30T13:12:19.468098+0000","last_undegraded":"2024-03-30T13:12:19.468098+0000","last_fullsized":"2024-03-30T13:12:19.468098+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:11:54.706131+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:11:54.706131+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:59.181418+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1139,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-30T13:12:19.797468+0000","last_change":"2024-03-30T13:11:54.781544+0000","last_active":"2024-03-30T13:12:19.797468+0000","last_peered":"2024-03-30T13:12:19.797468+0000","last_clean":"2024-03-30T13:12:19.797468+0000","last_became_active":"2024-03-30T13:11:09.212712+0000","last_became_peered":"2024-03-30T13:11:09.212712+0000","last_unstale":"2024-03-30T13:12:19.797468+0000","last_undegraded":"2024-03-30T13:12:19.797468+0000","last_fullsized":"2024-03-30T13:12:19.797468+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:54.781288+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:11:54.781288+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:21.599312+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":55832128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":858,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-30T13:12:19.386978+0000","last_change":"2024-03-30T13:12:05.457652+0000","last_active":"2024-03-30T13:12:19.386978+0000","last_peered":"2024-03-30T13:12:19.386978+0000","last_clean":"2024-03-30T13:12:19.386978+0000","last_became_active":"2024-03-30T13:12:00.203605+0000","last_became_peered":"2024-03-30T13:12:00.203605+0000","last_unstale":"2024-03-30T13:12:19.386978+0000","last_undegraded":"2024-03-30T13:12:19.386978+0000","last_fullsized":"2024-03-30T13:12:19.386978+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:05.457409+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:12:05.457409+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:32.879605+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1049,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-30T13:12:26.917752+0000","last_change":"2024-03-30T13:12:26.917834+0000","last_active":"2024-03-30T13:12:26.917752+0000","last_peered":"2024-03-30T13:12:26.917752+0000","last_clean":"2024-03-30T13:12:26.917752+0000","last_became_active":"2024-03-30T13:12:00.202952+0000","last_became_peered":"2024-03-30T13:12:00.202952+0000","last_unstale":"2024-03-30T13:12:26.917752+0000","last_undegraded":"2024-03-30T13:12:26.917752+0000","last_fullsized":"2024-03-30T13:12:26.917752+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:12:26.917553+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:56.679031+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1090,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-30T13:12:33.835608+0000","last_change":"2024-03-30T13:12:33.835689+0000","last_active":"2024-03-30T13:12:33.835608+0000","last_peered":"2024-03-30T13:12:33.835608+0000","last_clean":"2024-03-30T13:12:33.835608+0000","last_became_active":"2024-03-30T13:10:18.180670+0000","last_became_peered":"2024-03-30T13:10:18.180670+0000","last_unstale":"2024-03-30T13:12:33.835608+0000","last_undegraded":"2024-03-30T13:12:33.835608+0000","last_fullsized":"2024-03-30T13:12:33.835608+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:12:33.835431+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:12:33.835431+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:49.355370+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":399,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-30T13:12:36.623752+0000","last_change":"2024-03-30T13:12:36.623829+0000","last_active":"2024-03-30T13:12:36.623752+0000","last_peered":"2024-03-30T13:12:36.623752+0000","last_clean":"2024-03-30T13:12:36.623752+0000","last_became_active":"2024-03-30T13:10:18.183814+0000","last_became_peered":"2024-03-30T13:10:18.183814+0000","last_unstale":"2024-03-30T13:12:36.623752+0000","last_undegraded":"2024-03-30T13:12:36.623752+0000","last_fullsized":"2024-03-30T13:12:36.623752+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:12:36.623574+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:12:36.623574+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:58.642559+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":511,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-30T13:12:19.383662+0000","last_change":"2024-03-30T13:12:18.090983+0000","last_active":"2024-03-30T13:12:19.383662+0000","last_peered":"2024-03-30T13:12:19.383662+0000","last_clean":"2024-03-30T13:12:19.383662+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:12:19.383662+0000","last_undegraded":"2024-03-30T13:12:19.383662+0000","last_fullsized":"2024-03-30T13:12:19.383662+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:18.090642+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:12:18.090642+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:27.145345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":399,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-30T13:12:19.390462+0000","last_change":"2024-03-30T13:11:55.853801+0000","last_active":"2024-03-30T13:12:19.390462+0000","last_peered":"2024-03-30T13:12:19.390462+0000","last_clean":"2024-03-30T13:12:19.390462+0000","last_became_active":"2024-03-30T13:09:27.162655+0000","last_became_peered":"2024-03-30T13:09:27.162655+0000","last_unstale":"2024-03-30T13:12:19.390462+0000","last_undegraded":"2024-03-30T13:12:19.390462+0000","last_fullsized":"2024-03-30T13:12:19.390462+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:11:55.853519+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:11:55.853519+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:05.040598+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":17,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-03-30T13:12:19.436036+0000","last_change":"2024-03-30T13:12:19.435000+0000","last_active":"2024-03-30T13:12:19.436036+0000","last_peered":"2024-03-30T13:12:19.436036+0000","last_clean":"2024-03-30T13:12:19.436036+0000","last_became_active":"2024-03-30T13:12:19.393644+0000","last_became_peered":"2024-03-30T13:12:19.393644+0000","last_unstale":"2024-03-30T13:12:19.436036+0000","last_undegraded":"2024-03-30T13:12:19.436036+0000","last_fullsized":"2024-03-30T13:12:19.436036+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:19.631559+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:11:19.631559+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:12:45.502284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5226911,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":232219217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102486016,"data_stored":102486016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834153,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52704,"kb_used_data":20568,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94319136,"statfs":{"total":96636764160,"available":96582795264,"internally_reserved":0,"allocated":21061632,"data_stored":17985188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534781,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62056,"kb_used_data":25312,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94309784,"statfs":{"total":96636764160,"available":96573218816,"internally_reserved":0,"allocated":25919488,"data_stored":22843044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":37609960},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":441,"seq":1894080577546,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56164,"kb_used_data":18968,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94315676,"statfs":{"total":96636764160,"available":96579252224,"internally_reserved":0,"allocated":19423232,"data_stored":16346788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9343,"internal_metadata":38067073},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509649,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46616,"kb_used_data":15432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325224,"statfs":{"total":96636764160,"available":96589029376,"internally_reserved":0,"allocated":15802368,"data_stored":12725924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509650,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50136,"kb_used_data":18960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321704,"statfs":{"total":96636764160,"available":96585424896,"internally_reserved":0,"allocated":19415040,"data_stored":16338596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297044,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53144,"kb_used_data":15504,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94318696,"statfs":{"total":96636764160,"available":96582344704,"internally_reserved":0,"allocated":15876096,"data_stored":12799652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147358,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49648,"kb_used_data":17960,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94322192,"statfs":{"total":96636764160,"available":96585924608,"internally_reserved":0,"allocated":18391040,"data_stored":15314596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10211328,"data_stored":10211328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":13832192,"data_stored":13832192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:12:44.052 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:12:44.052 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:12:44.052 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-03-30T13:12:44.052 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-30T13:12:44.052 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-30T13:12:44.054 INFO:tasks.ceph.osd.1:Started 2024-03-30T13:12:44.055 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-30T13:12:44.109 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:12:44.110+0000 7fb2e87bd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:44.120 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:12:44.122+0000 7fb2e87bd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:44.120 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:12:44.122+0000 7fb2e87bd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:44.179 INFO:teuthology.orchestra.run.smithi067.stderr:no valid command found; 10 closest matches: 2024-03-30T13:12:44.179 INFO:teuthology.orchestra.run.smithi067.stderr:0 2024-03-30T13:12:44.179 INFO:teuthology.orchestra.run.smithi067.stderr:1 2024-03-30T13:12:44.179 INFO:teuthology.orchestra.run.smithi067.stderr:2 2024-03-30T13:12:44.179 INFO:teuthology.orchestra.run.smithi067.stderr:abort 2024-03-30T13:12:44.179 INFO:teuthology.orchestra.run.smithi067.stderr:assert 2024-03-30T13:12:44.179 INFO:teuthology.orchestra.run.smithi067.stderr:config diff 2024-03-30T13:12:44.179 INFO:teuthology.orchestra.run.smithi067.stderr:config diff get 2024-03-30T13:12:44.179 INFO:teuthology.orchestra.run.smithi067.stderr:config get 2024-03-30T13:12:44.179 INFO:teuthology.orchestra.run.smithi067.stderr:config help [] 2024-03-30T13:12:44.180 INFO:teuthology.orchestra.run.smithi067.stderr:config set ... 2024-03-30T13:12:44.180 INFO:teuthology.orchestra.run.smithi067.stderr:admin_socket: invalid command 2024-03-30T13:12:44.181 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:12:44.182 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-30T13:12:44.658 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:12:44.658+0000 7fb2e87bd7c0 -1 Falling back to public interface 2024-03-30T13:12:48.424 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:12:48.425+0000 7fb2e87bd7c0 -1 osd.1 457 log_to_monitors true 2024-03-30T13:12:49.183 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-30T13:12:49.222 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:12:49.221+0000 7fb2dc58a640 -1 osd.1 457 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:12:49.315 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-30T13:12:49.316 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-30T13:12:49.316 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-03-30T13:12:49.316 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-30T13:12:49.329 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:12:49.329 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-03-30T13:12:49.330 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 1 2024-03-30T13:12:49.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:49.493+0000 7f3e98128640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:49.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:49.509+0000 7f3e98128640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:49.691 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:12:49.689+0000 7fbb38f08640 -1 mon.a@0(leader).osd e460 definitely_dead 0 2024-03-30T13:12:50.193 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.1. 2024-03-30T13:12:50.224 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-03-30T13:12:50.224 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:12:50.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:50.337+0000 7ff94a180640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:50.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:50.353+0000 7ff94a180640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:51.195 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:12:51.227 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:12:51.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:51.349+0000 7f9355e3e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:51.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:51.365+0000 7f9355e3e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:52.218 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:12:52.243 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:12:52.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:52.357+0000 7f46e39be640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:52.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:52.373+0000 7f46e39be640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:53.223 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:12:53.249 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:12:53.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:53.369+0000 7f8dde1e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:53.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:53.385+0000 7f8dde1e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:54.233 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:12:54.259 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:12:54.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:54.377+0000 7f542648e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:54.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:54.393+0000 7f542648e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:55.235 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:12:55.266 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:12:55.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:55.385+0000 7ff3541d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:55.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:55.401+0000 7ff3541d6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:56.264 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:12:56.286 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:12:56.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:56.401+0000 7fbf848e0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:56.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:56.421+0000 7fbf848e0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:57.257 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:12:57.283 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:12:57.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:57.401+0000 7f6c805f2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:57.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:57.417+0000 7f6c805f2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:58.268 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:12:58.294 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:12:58.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:58.405+0000 7f86ca565640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:58.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:58.425+0000 7f86ca565640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:59.281 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:12:59.307 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:12:59.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:59.429+0000 7f7483a9e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:12:59.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:12:59.445+0000 7f7483a9e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:00.291 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:13:00.319 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:13:00.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:00.437+0000 7fd7c4cee640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:00.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:00.453+0000 7fd7c4cee640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:01.304 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:13:01.329 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:13:01.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:01.445+0000 7ffb11b85640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:01.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:01.465+0000 7ffb11b85640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:02.317 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:13:02.342 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:13:02.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:02.461+0000 7fa3db614640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:02.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:02.477+0000 7fa3db614640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:03.342 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:13:03.369 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:13:03.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:03.489+0000 7f220699f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:03.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:03.505+0000 7f220699f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:04.344 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:13:04.372 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:13:04.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:04.489+0000 7f1ea5a97640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:04.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:04.509+0000 7f1ea5a97640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:05.366 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:13:05.390 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:13:05.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:05.509+0000 7f545ec60640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:05.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:05.525+0000 7f545ec60640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:06.383 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:13:06.408 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:13:06.408 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:13:06.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:06.529+0000 7f52ba9da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:06.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:06.545+0000 7f52ba9da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:06.716 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:13:06.716 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:13:06.734 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":997,"stamp":"2024-03-30T13:13:06.386190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":226992306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":414116,"kb_used_data":139556,"kb_used_omap":49,"kb_used_meta":274382,"kb_avail":754560604,"statfs":{"total":773094113280,"available":772670058496,"internally_reserved":0,"allocated":142905344,"data_stored":117517180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":51011,"internal_metadata":280967357},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.064836"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1067,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-03-30T13:13:05.850815+0000","last_change":"2024-03-30T13:12:53.896798+0000","last_active":"2024-03-30T13:13:05.850815+0000","last_peered":"2024-03-30T13:13:05.850815+0000","last_clean":"2024-03-30T13:13:05.850815+0000","last_became_active":"2024-03-30T13:12:00.205621+0000","last_became_peered":"2024-03-30T13:12:00.205621+0000","last_unstale":"2024-03-30T13:13:05.850815+0000","last_undegraded":"2024-03-30T13:13:05.850815+0000","last_fullsized":"2024-03-30T13:13:05.850815+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:12:53.896535+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:12:53.896535+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:04.066529+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1014,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-30T13:13:03.348743+0000","last_change":"2024-03-30T13:12:09.454216+0000","last_active":"2024-03-30T13:13:03.348743+0000","last_peered":"2024-03-30T13:13:03.348743+0000","last_clean":"2024-03-30T13:13:03.348743+0000","last_became_active":"2024-03-30T13:12:00.216256+0000","last_became_peered":"2024-03-30T13:12:00.216256+0000","last_unstale":"2024-03-30T13:13:03.348743+0000","last_undegraded":"2024-03-30T13:13:03.348743+0000","last_fullsized":"2024-03-30T13:13:03.348743+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:09.453958+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_clean_scrub_stamp":"2024-03-30T13:12:09.453958+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:33.947990+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":626,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-30T13:13:02.325743+0000","last_change":"2024-03-30T13:12:51.213469+0000","last_active":"2024-03-30T13:13:02.325743+0000","last_peered":"2024-03-30T13:13:02.325743+0000","last_clean":"2024-03-30T13:13:02.325743+0000","last_became_active":"2024-03-30T13:12:51.212130+0000","last_became_peered":"2024-03-30T13:12:51.212130+0000","last_unstale":"2024-03-30T13:13:02.325743+0000","last_undegraded":"2024-03-30T13:13:02.325743+0000","last_fullsized":"2024-03-30T13:13:02.325743+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:12:00.319122+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:12:00.319122+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:26.588253+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":568,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-30T13:13:02.325684+0000","last_change":"2024-03-30T13:12:23.100937+0000","last_active":"2024-03-30T13:13:02.325684+0000","last_peered":"2024-03-30T13:13:02.325684+0000","last_clean":"2024-03-30T13:13:02.325684+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:13:02.325684+0000","last_undegraded":"2024-03-30T13:13:02.325684+0000","last_fullsized":"2024-03-30T13:13:02.325684+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:12:23.100675+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:12:23.100675+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:44.577727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":387,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-30T13:13:02.323768+0000","last_change":"2024-03-30T13:12:20.783890+0000","last_active":"2024-03-30T13:13:02.323768+0000","last_peered":"2024-03-30T13:13:02.323768+0000","last_clean":"2024-03-30T13:13:02.323768+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:13:02.323768+0000","last_undegraded":"2024-03-30T13:13:02.323768+0000","last_fullsized":"2024-03-30T13:13:02.323768+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:20.783652+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:12:20.783652+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:32.641222+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1044,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-03-30T13:13:01.635362+0000","last_change":"2024-03-30T13:12:15.741547+0000","last_active":"2024-03-30T13:13:01.635362+0000","last_peered":"2024-03-30T13:13:01.635362+0000","last_clean":"2024-03-30T13:13:01.635362+0000","last_became_active":"2024-03-30T13:10:18.181149+0000","last_became_peered":"2024-03-30T13:10:18.181149+0000","last_unstale":"2024-03-30T13:13:01.635362+0000","last_undegraded":"2024-03-30T13:13:01.635362+0000","last_fullsized":"2024-03-30T13:13:01.635362+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:12:15.741379+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:36.529709+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":603,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-30T13:13:02.325717+0000","last_change":"2024-03-30T13:12:04.242615+0000","last_active":"2024-03-30T13:13:02.325717+0000","last_peered":"2024-03-30T13:13:02.325717+0000","last_clean":"2024-03-30T13:13:02.325717+0000","last_became_active":"2024-03-30T13:12:00.202949+0000","last_became_peered":"2024-03-30T13:12:00.202949+0000","last_unstale":"2024-03-30T13:13:02.325717+0000","last_undegraded":"2024-03-30T13:13:02.325717+0000","last_fullsized":"2024-03-30T13:13:02.325717+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:12:04.242418+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:12:04.242418+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:12.083440+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":321,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-03-30T13:13:04.352206+0000","last_change":"2024-03-30T13:12:51.513374+0000","last_active":"2024-03-30T13:13:04.352206+0000","last_peered":"2024-03-30T13:13:04.352206+0000","last_clean":"2024-03-30T13:13:04.352206+0000","last_became_active":"2024-03-30T13:12:51.512874+0000","last_became_peered":"2024-03-30T13:12:51.512874+0000","last_unstale":"2024-03-30T13:13:04.352206+0000","last_undegraded":"2024-03-30T13:13:04.352206+0000","last_fullsized":"2024-03-30T13:13:04.352206+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:11:54.706131+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:11:54.706131+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:05.329049+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1162,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-03-30T13:13:01.635226+0000","last_change":"2024-03-30T13:11:54.781544+0000","last_active":"2024-03-30T13:13:01.635226+0000","last_peered":"2024-03-30T13:13:01.635226+0000","last_clean":"2024-03-30T13:13:01.635226+0000","last_became_active":"2024-03-30T13:11:09.212712+0000","last_became_peered":"2024-03-30T13:11:09.212712+0000","last_unstale":"2024-03-30T13:13:01.635226+0000","last_undegraded":"2024-03-30T13:13:01.635226+0000","last_fullsized":"2024-03-30T13:13:01.635226+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:54.781288+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:11:54.781288+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:21.599312+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":55832128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":888,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-30T13:13:03.348682+0000","last_change":"2024-03-30T13:12:05.457652+0000","last_active":"2024-03-30T13:13:03.348682+0000","last_peered":"2024-03-30T13:13:03.348682+0000","last_clean":"2024-03-30T13:13:03.348682+0000","last_became_active":"2024-03-30T13:12:00.203605+0000","last_became_peered":"2024-03-30T13:12:00.203605+0000","last_unstale":"2024-03-30T13:13:03.348682+0000","last_undegraded":"2024-03-30T13:13:03.348682+0000","last_fullsized":"2024-03-30T13:13:03.348682+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:05.457409+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:12:05.457409+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:32.879605+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1080,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-03-30T13:13:05.850760+0000","last_change":"2024-03-30T13:12:26.917834+0000","last_active":"2024-03-30T13:13:05.850760+0000","last_peered":"2024-03-30T13:13:05.850760+0000","last_clean":"2024-03-30T13:13:05.850760+0000","last_became_active":"2024-03-30T13:12:00.202952+0000","last_became_peered":"2024-03-30T13:12:00.202952+0000","last_unstale":"2024-03-30T13:13:05.850760+0000","last_undegraded":"2024-03-30T13:13:05.850760+0000","last_fullsized":"2024-03-30T13:13:05.850760+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:12:26.917553+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:56.679031+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1121,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-03-30T13:13:05.850761+0000","last_change":"2024-03-30T13:12:33.835689+0000","last_active":"2024-03-30T13:13:05.850761+0000","last_peered":"2024-03-30T13:13:05.850761+0000","last_clean":"2024-03-30T13:13:05.850761+0000","last_became_active":"2024-03-30T13:10:18.180670+0000","last_became_peered":"2024-03-30T13:10:18.180670+0000","last_unstale":"2024-03-30T13:13:05.850761+0000","last_undegraded":"2024-03-30T13:13:05.850761+0000","last_fullsized":"2024-03-30T13:13:05.850761+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:12:33.835431+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:12:33.835431+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:49.355370+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":425,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-30T13:13:02.323761+0000","last_change":"2024-03-30T13:12:36.623829+0000","last_active":"2024-03-30T13:13:02.323761+0000","last_peered":"2024-03-30T13:13:02.323761+0000","last_clean":"2024-03-30T13:13:02.323761+0000","last_became_active":"2024-03-30T13:10:18.183814+0000","last_became_peered":"2024-03-30T13:10:18.183814+0000","last_unstale":"2024-03-30T13:13:02.323761+0000","last_undegraded":"2024-03-30T13:13:02.323761+0000","last_fullsized":"2024-03-30T13:13:02.323761+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:12:36.623574+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:12:36.623574+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:58.642559+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":539,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-30T13:13:02.325684+0000","last_change":"2024-03-30T13:12:18.090983+0000","last_active":"2024-03-30T13:13:02.325684+0000","last_peered":"2024-03-30T13:13:02.325684+0000","last_clean":"2024-03-30T13:13:02.325684+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:13:02.325684+0000","last_undegraded":"2024-03-30T13:13:02.325684+0000","last_fullsized":"2024-03-30T13:13:02.325684+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:18.090642+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:12:18.090642+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:27.145345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":425,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-30T13:13:02.324033+0000","last_change":"2024-03-30T13:11:55.853801+0000","last_active":"2024-03-30T13:13:02.324033+0000","last_peered":"2024-03-30T13:13:02.324033+0000","last_clean":"2024-03-30T13:13:02.324033+0000","last_became_active":"2024-03-30T13:09:27.162655+0000","last_became_peered":"2024-03-30T13:09:27.162655+0000","last_unstale":"2024-03-30T13:13:02.324033+0000","last_undegraded":"2024-03-30T13:13:02.324033+0000","last_fullsized":"2024-03-30T13:13:02.324033+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:11:55.853519+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:11:55.853519+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:05.040598+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":278,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-03-30T13:13:04.352217+0000","last_change":"2024-03-30T13:12:51.515035+0000","last_active":"2024-03-30T13:13:04.352217+0000","last_peered":"2024-03-30T13:13:04.352217+0000","last_clean":"2024-03-30T13:13:04.352217+0000","last_became_active":"2024-03-30T13:12:51.513803+0000","last_became_peered":"2024-03-30T13:12:51.513803+0000","last_unstale":"2024-03-30T13:13:04.352217+0000","last_undegraded":"2024-03-30T13:13:04.352217+0000","last_fullsized":"2024-03-30T13:13:04.352217+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:12:45.634758+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:12:45.634758+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:05.867044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":226992306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834158,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52900,"kb_used_data":20760,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94318940,"statfs":{"total":96636764160,"available":96582594560,"internally_reserved":0,"allocated":21258240,"data_stored":18074518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534786,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62232,"kb_used_data":25480,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94309608,"statfs":{"total":96636764160,"available":96573038592,"internally_reserved":0,"allocated":26091520,"data_stored":22921400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":37609960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":441,"seq":1894080577551,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54400,"kb_used_data":17196,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94317440,"statfs":{"total":96636764160,"available":96581058560,"internally_reserved":0,"allocated":17608704,"data_stored":14431783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9343,"internal_metadata":38067073},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509654,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47300,"kb_used_data":11952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94324540,"statfs":{"total":96636764160,"available":96588328960,"internally_reserved":0,"allocated":12238848,"data_stored":9068728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":461,"seq":1979979923460,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43840,"kb_used_data":11252,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94328000,"statfs":{"total":96636764160,"available":96591872000,"internally_reserved":0,"allocated":11522048,"data_stored":8338326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7692,"internal_metadata":33350132},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509655,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50316,"kb_used_data":19128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321524,"statfs":{"total":96636764160,"available":96585240576,"internally_reserved":0,"allocated":19587072,"data_stored":16416952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297049,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53320,"kb_used_data":15672,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94318520,"statfs":{"total":96636764160,"available":96582164480,"internally_reserved":0,"allocated":16048128,"data_stored":12878008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147363,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49808,"kb_used_data":18116,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94322032,"statfs":{"total":96636764160,"available":96585760768,"internally_reserved":0,"allocated":18550784,"data_stored":15387465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:13:06.736 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:13:06.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:06.853+0000 7f8f80d1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:06.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:06.865+0000 7f8f80d1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:07.043 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:13:07.043 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:13:07.062 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":997,"stamp":"2024-03-30T13:13:06.386190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":226992306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":414116,"kb_used_data":139556,"kb_used_omap":49,"kb_used_meta":274382,"kb_avail":754560604,"statfs":{"total":773094113280,"available":772670058496,"internally_reserved":0,"allocated":142905344,"data_stored":117517180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":51011,"internal_metadata":280967357},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.064836"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1067,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-03-30T13:13:05.850815+0000","last_change":"2024-03-30T13:12:53.896798+0000","last_active":"2024-03-30T13:13:05.850815+0000","last_peered":"2024-03-30T13:13:05.850815+0000","last_clean":"2024-03-30T13:13:05.850815+0000","last_became_active":"2024-03-30T13:12:00.205621+0000","last_became_peered":"2024-03-30T13:12:00.205621+0000","last_unstale":"2024-03-30T13:13:05.850815+0000","last_undegraded":"2024-03-30T13:13:05.850815+0000","last_fullsized":"2024-03-30T13:13:05.850815+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:12:53.896535+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:12:53.896535+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:04.066529+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1014,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-30T13:13:03.348743+0000","last_change":"2024-03-30T13:12:09.454216+0000","last_active":"2024-03-30T13:13:03.348743+0000","last_peered":"2024-03-30T13:13:03.348743+0000","last_clean":"2024-03-30T13:13:03.348743+0000","last_became_active":"2024-03-30T13:12:00.216256+0000","last_became_peered":"2024-03-30T13:12:00.216256+0000","last_unstale":"2024-03-30T13:13:03.348743+0000","last_undegraded":"2024-03-30T13:13:03.348743+0000","last_fullsized":"2024-03-30T13:13:03.348743+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:09.453958+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_clean_scrub_stamp":"2024-03-30T13:12:09.453958+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:33.947990+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":626,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-30T13:13:02.325743+0000","last_change":"2024-03-30T13:12:51.213469+0000","last_active":"2024-03-30T13:13:02.325743+0000","last_peered":"2024-03-30T13:13:02.325743+0000","last_clean":"2024-03-30T13:13:02.325743+0000","last_became_active":"2024-03-30T13:12:51.212130+0000","last_became_peered":"2024-03-30T13:12:51.212130+0000","last_unstale":"2024-03-30T13:13:02.325743+0000","last_undegraded":"2024-03-30T13:13:02.325743+0000","last_fullsized":"2024-03-30T13:13:02.325743+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:12:00.319122+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:12:00.319122+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:26.588253+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":568,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-30T13:13:02.325684+0000","last_change":"2024-03-30T13:12:23.100937+0000","last_active":"2024-03-30T13:13:02.325684+0000","last_peered":"2024-03-30T13:13:02.325684+0000","last_clean":"2024-03-30T13:13:02.325684+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:13:02.325684+0000","last_undegraded":"2024-03-30T13:13:02.325684+0000","last_fullsized":"2024-03-30T13:13:02.325684+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:12:23.100675+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:12:23.100675+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:44.577727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":387,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-30T13:13:02.323768+0000","last_change":"2024-03-30T13:12:20.783890+0000","last_active":"2024-03-30T13:13:02.323768+0000","last_peered":"2024-03-30T13:13:02.323768+0000","last_clean":"2024-03-30T13:13:02.323768+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:13:02.323768+0000","last_undegraded":"2024-03-30T13:13:02.323768+0000","last_fullsized":"2024-03-30T13:13:02.323768+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:20.783652+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:12:20.783652+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:32.641222+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1044,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-03-30T13:13:01.635362+0000","last_change":"2024-03-30T13:12:15.741547+0000","last_active":"2024-03-30T13:13:01.635362+0000","last_peered":"2024-03-30T13:13:01.635362+0000","last_clean":"2024-03-30T13:13:01.635362+0000","last_became_active":"2024-03-30T13:10:18.181149+0000","last_became_peered":"2024-03-30T13:10:18.181149+0000","last_unstale":"2024-03-30T13:13:01.635362+0000","last_undegraded":"2024-03-30T13:13:01.635362+0000","last_fullsized":"2024-03-30T13:13:01.635362+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:12:15.741379+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:36.529709+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":603,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-30T13:13:02.325717+0000","last_change":"2024-03-30T13:12:04.242615+0000","last_active":"2024-03-30T13:13:02.325717+0000","last_peered":"2024-03-30T13:13:02.325717+0000","last_clean":"2024-03-30T13:13:02.325717+0000","last_became_active":"2024-03-30T13:12:00.202949+0000","last_became_peered":"2024-03-30T13:12:00.202949+0000","last_unstale":"2024-03-30T13:13:02.325717+0000","last_undegraded":"2024-03-30T13:13:02.325717+0000","last_fullsized":"2024-03-30T13:13:02.325717+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:12:04.242418+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:12:04.242418+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:12.083440+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":321,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-03-30T13:13:04.352206+0000","last_change":"2024-03-30T13:12:51.513374+0000","last_active":"2024-03-30T13:13:04.352206+0000","last_peered":"2024-03-30T13:13:04.352206+0000","last_clean":"2024-03-30T13:13:04.352206+0000","last_became_active":"2024-03-30T13:12:51.512874+0000","last_became_peered":"2024-03-30T13:12:51.512874+0000","last_unstale":"2024-03-30T13:13:04.352206+0000","last_undegraded":"2024-03-30T13:13:04.352206+0000","last_fullsized":"2024-03-30T13:13:04.352206+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:11:54.706131+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:11:54.706131+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:05.329049+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1162,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-03-30T13:13:01.635226+0000","last_change":"2024-03-30T13:11:54.781544+0000","last_active":"2024-03-30T13:13:01.635226+0000","last_peered":"2024-03-30T13:13:01.635226+0000","last_clean":"2024-03-30T13:13:01.635226+0000","last_became_active":"2024-03-30T13:11:09.212712+0000","last_became_peered":"2024-03-30T13:11:09.212712+0000","last_unstale":"2024-03-30T13:13:01.635226+0000","last_undegraded":"2024-03-30T13:13:01.635226+0000","last_fullsized":"2024-03-30T13:13:01.635226+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:54.781288+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:11:54.781288+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:21.599312+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":55832128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":888,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-30T13:13:03.348682+0000","last_change":"2024-03-30T13:12:05.457652+0000","last_active":"2024-03-30T13:13:03.348682+0000","last_peered":"2024-03-30T13:13:03.348682+0000","last_clean":"2024-03-30T13:13:03.348682+0000","last_became_active":"2024-03-30T13:12:00.203605+0000","last_became_peered":"2024-03-30T13:12:00.203605+0000","last_unstale":"2024-03-30T13:13:03.348682+0000","last_undegraded":"2024-03-30T13:13:03.348682+0000","last_fullsized":"2024-03-30T13:13:03.348682+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:05.457409+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:12:05.457409+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:32.879605+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1080,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-03-30T13:13:05.850760+0000","last_change":"2024-03-30T13:12:26.917834+0000","last_active":"2024-03-30T13:13:05.850760+0000","last_peered":"2024-03-30T13:13:05.850760+0000","last_clean":"2024-03-30T13:13:05.850760+0000","last_became_active":"2024-03-30T13:12:00.202952+0000","last_became_peered":"2024-03-30T13:12:00.202952+0000","last_unstale":"2024-03-30T13:13:05.850760+0000","last_undegraded":"2024-03-30T13:13:05.850760+0000","last_fullsized":"2024-03-30T13:13:05.850760+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:12:26.917553+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:56.679031+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1121,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-03-30T13:13:05.850761+0000","last_change":"2024-03-30T13:12:33.835689+0000","last_active":"2024-03-30T13:13:05.850761+0000","last_peered":"2024-03-30T13:13:05.850761+0000","last_clean":"2024-03-30T13:13:05.850761+0000","last_became_active":"2024-03-30T13:10:18.180670+0000","last_became_peered":"2024-03-30T13:10:18.180670+0000","last_unstale":"2024-03-30T13:13:05.850761+0000","last_undegraded":"2024-03-30T13:13:05.850761+0000","last_fullsized":"2024-03-30T13:13:05.850761+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:12:33.835431+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:12:33.835431+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:49.355370+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":425,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-30T13:13:02.323761+0000","last_change":"2024-03-30T13:12:36.623829+0000","last_active":"2024-03-30T13:13:02.323761+0000","last_peered":"2024-03-30T13:13:02.323761+0000","last_clean":"2024-03-30T13:13:02.323761+0000","last_became_active":"2024-03-30T13:10:18.183814+0000","last_became_peered":"2024-03-30T13:10:18.183814+0000","last_unstale":"2024-03-30T13:13:02.323761+0000","last_undegraded":"2024-03-30T13:13:02.323761+0000","last_fullsized":"2024-03-30T13:13:02.323761+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:12:36.623574+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:12:36.623574+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:58.642559+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":539,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-30T13:13:02.325684+0000","last_change":"2024-03-30T13:12:18.090983+0000","last_active":"2024-03-30T13:13:02.325684+0000","last_peered":"2024-03-30T13:13:02.325684+0000","last_clean":"2024-03-30T13:13:02.325684+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:13:02.325684+0000","last_undegraded":"2024-03-30T13:13:02.325684+0000","last_fullsized":"2024-03-30T13:13:02.325684+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:18.090642+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:12:18.090642+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:27.145345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":425,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-30T13:13:02.324033+0000","last_change":"2024-03-30T13:11:55.853801+0000","last_active":"2024-03-30T13:13:02.324033+0000","last_peered":"2024-03-30T13:13:02.324033+0000","last_clean":"2024-03-30T13:13:02.324033+0000","last_became_active":"2024-03-30T13:09:27.162655+0000","last_became_peered":"2024-03-30T13:09:27.162655+0000","last_unstale":"2024-03-30T13:13:02.324033+0000","last_undegraded":"2024-03-30T13:13:02.324033+0000","last_fullsized":"2024-03-30T13:13:02.324033+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:11:55.853519+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:11:55.853519+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:05.040598+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":278,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-03-30T13:13:04.352217+0000","last_change":"2024-03-30T13:12:51.515035+0000","last_active":"2024-03-30T13:13:04.352217+0000","last_peered":"2024-03-30T13:13:04.352217+0000","last_clean":"2024-03-30T13:13:04.352217+0000","last_became_active":"2024-03-30T13:12:51.513803+0000","last_became_peered":"2024-03-30T13:12:51.513803+0000","last_unstale":"2024-03-30T13:13:04.352217+0000","last_undegraded":"2024-03-30T13:13:04.352217+0000","last_fullsized":"2024-03-30T13:13:04.352217+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:12:45.634758+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:12:45.634758+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:05.867044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":226992306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834158,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52900,"kb_used_data":20760,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94318940,"statfs":{"total":96636764160,"available":96582594560,"internally_reserved":0,"allocated":21258240,"data_stored":18074518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534786,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62232,"kb_used_data":25480,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94309608,"statfs":{"total":96636764160,"available":96573038592,"internally_reserved":0,"allocated":26091520,"data_stored":22921400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":37609960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":441,"seq":1894080577551,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54400,"kb_used_data":17196,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94317440,"statfs":{"total":96636764160,"available":96581058560,"internally_reserved":0,"allocated":17608704,"data_stored":14431783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9343,"internal_metadata":38067073},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509654,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47300,"kb_used_data":11952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94324540,"statfs":{"total":96636764160,"available":96588328960,"internally_reserved":0,"allocated":12238848,"data_stored":9068728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":461,"seq":1979979923460,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43840,"kb_used_data":11252,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94328000,"statfs":{"total":96636764160,"available":96591872000,"internally_reserved":0,"allocated":11522048,"data_stored":8338326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7692,"internal_metadata":33350132},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509655,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50316,"kb_used_data":19128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321524,"statfs":{"total":96636764160,"available":96585240576,"internally_reserved":0,"allocated":19587072,"data_stored":16416952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297049,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53320,"kb_used_data":15672,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94318520,"statfs":{"total":96636764160,"available":96582164480,"internally_reserved":0,"allocated":16048128,"data_stored":12878008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147363,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49808,"kb_used_data":18116,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94322032,"statfs":{"total":96636764160,"available":96585760768,"internally_reserved":0,"allocated":18550784,"data_stored":15387465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:13:07.063 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:13:07.063 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:13:07.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:07.173+0000 7f1047634640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:07.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:07.185+0000 7f1047634640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:07.356 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:13:07.357 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":477,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:13:06.370455+0000","last_up_change":"2024-03-30T13:12:50.192483+0000","last_in_change":"2024-03-30T13:12:50.192483+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":25,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":436,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":346,"last_clean_end":457,"up_from":461,"up_thru":461,"down_at":458,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2034332759},{"type":"v1","addr":"172.21.15.67:6818","nonce":2034332759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2034332759},{"type":"v1","addr":"172.21.15.67:6821","nonce":2034332759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2034332759},{"type":"v1","addr":"172.21.15.67:6828","nonce":2034332759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2034332759},{"type":"v1","addr":"172.21.15.67:6825","nonce":2034332759}]},"public_addr":"172.21.15.67:6818/2034332759","cluster_addr":"172.21.15.67:6821/2034332759","heartbeat_back_addr":"172.21.15.67:6828/2034332759","heartbeat_front_addr":"172.21.15.67:6825/2034332759","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":458,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":388,"up_from":393,"up_thru":419,"down_at":389,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6801","nonce":3837194718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6803","nonce":3837194718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6807","nonce":3837194718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6805","nonce":3837194718}]},"public_addr":"172.21.15.67:6801/3837194718","cluster_addr":"172.21.15.67:6803/3837194718","heartbeat_back_addr":"172.21.15.67:6807/3837194718","heartbeat_front_addr":"172.21.15.67:6805/3837194718","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":250,"last_clean_end":435,"up_from":441,"up_thru":458,"down_at":436,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3152653542},{"type":"v1","addr":"172.21.15.82:6803","nonce":3152653542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":3152653542},{"type":"v1","addr":"172.21.15.82:6807","nonce":3152653542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":3152653542},{"type":"v1","addr":"172.21.15.82:6815","nonce":3152653542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":3152653542},{"type":"v1","addr":"172.21.15.82:6811","nonce":3152653542}]},"public_addr":"172.21.15.82:6803/3152653542","cluster_addr":"172.21.15.82:6807/3152653542","heartbeat_back_addr":"172.21.15.82:6815/3152653542","heartbeat_front_addr":"172.21.15.82:6811/3152653542","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":274,"last_clean_end":314,"up_from":320,"up_thru":411,"down_at":315,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":925466951},{"type":"v1","addr":"172.21.15.82:6817","nonce":925466951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":925466951},{"type":"v1","addr":"172.21.15.82:6819","nonce":925466951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":925466951},{"type":"v1","addr":"172.21.15.82:6823","nonce":925466951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":925466951},{"type":"v1","addr":"172.21.15.82:6821","nonce":925466951}]},"public_addr":"172.21.15.82:6817/925466951","cluster_addr":"172.21.15.82:6819/925466951","heartbeat_back_addr":"172.21.15.82:6823/925466951","heartbeat_front_addr":"172.21.15.82:6821/925466951","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":227,"last_clean_end":363,"up_from":372,"up_thru":441,"down_at":364,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6801","nonce":189641108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6806","nonce":189641108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6814","nonce":189641108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6810","nonce":189641108}]},"public_addr":"172.21.15.82:6801/189641108","cluster_addr":"172.21.15.82:6806/189641108","heartbeat_back_addr":"172.21.15.82:6814/189641108","heartbeat_front_addr":"172.21.15.82:6810/189641108","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":298,"last_clean_end":409,"up_from":419,"up_thru":461,"down_at":410,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6825","nonce":2166599911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6827","nonce":2166599911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6831","nonce":2166599911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6829","nonce":2166599911}]},"public_addr":"172.21.15.82:6825/2166599911","cluster_addr":"172.21.15.82:6827/2166599911","heartbeat_back_addr":"172.21.15.82:6831/2166599911","heartbeat_front_addr":"172.21.15.82:6829/2166599911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:12:04.096476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:12:03.772581+0000","dead_epoch":458},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:12:09.071073+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:11:43.966354+0000","dead_epoch":389},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:11:58.441017+0000","dead_epoch":436},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:12:08.962115+0000","dead_epoch":315},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:13:00.706194+0000","dead_epoch":364},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:12:04.878998+0000","dead_epoch":410}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:13:07.374 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:13:07.375 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:13:07.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:07.481+0000 7fc6e255d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:07.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:07.497+0000 7fc6e255d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:07.664 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:13:07.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:13:07.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:07.793+0000 7f5f588b9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:07.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:07.809+0000 7f5f588b9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:07.982 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:13:07.982 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:13:07.999 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:13:08.000 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:13:08.000 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:13:08.000 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:13:08.000 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 5, 6, 3, 7, 4, 1] 2024-03-30T13:13:08.000 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:13:08.000 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:13:08.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:08.109+0000 7ff3a2dcf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:08.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:08.121+0000 7ff3a2dcf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:08.289 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:13:08.289 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:13:08.306 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":999,"stamp":"2024-03-30T13:13:08.086267+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":226992306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":414168,"kb_used_data":139604,"kb_used_omap":49,"kb_used_meta":274382,"kb_avail":754560552,"statfs":{"total":773094113280,"available":772670005248,"internally_reserved":0,"allocated":142954496,"data_stored":117539128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":51011,"internal_metadata":280967357},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000993"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1067,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-03-30T13:13:05.850815+0000","last_change":"2024-03-30T13:12:53.896798+0000","last_active":"2024-03-30T13:13:05.850815+0000","last_peered":"2024-03-30T13:13:05.850815+0000","last_clean":"2024-03-30T13:13:05.850815+0000","last_became_active":"2024-03-30T13:12:00.205621+0000","last_became_peered":"2024-03-30T13:12:00.205621+0000","last_unstale":"2024-03-30T13:13:05.850815+0000","last_undegraded":"2024-03-30T13:13:05.850815+0000","last_fullsized":"2024-03-30T13:13:05.850815+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:12:53.896535+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:12:53.896535+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:04.066529+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1014,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-30T13:13:03.348743+0000","last_change":"2024-03-30T13:12:09.454216+0000","last_active":"2024-03-30T13:13:03.348743+0000","last_peered":"2024-03-30T13:13:03.348743+0000","last_clean":"2024-03-30T13:13:03.348743+0000","last_became_active":"2024-03-30T13:12:00.216256+0000","last_became_peered":"2024-03-30T13:12:00.216256+0000","last_unstale":"2024-03-30T13:13:03.348743+0000","last_undegraded":"2024-03-30T13:13:03.348743+0000","last_fullsized":"2024-03-30T13:13:03.348743+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:09.453958+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_clean_scrub_stamp":"2024-03-30T13:12:09.453958+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:33.947990+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":626,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-30T13:13:02.325743+0000","last_change":"2024-03-30T13:12:51.213469+0000","last_active":"2024-03-30T13:13:02.325743+0000","last_peered":"2024-03-30T13:13:02.325743+0000","last_clean":"2024-03-30T13:13:02.325743+0000","last_became_active":"2024-03-30T13:12:51.212130+0000","last_became_peered":"2024-03-30T13:12:51.212130+0000","last_unstale":"2024-03-30T13:13:02.325743+0000","last_undegraded":"2024-03-30T13:13:02.325743+0000","last_fullsized":"2024-03-30T13:13:02.325743+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:12:00.319122+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:12:00.319122+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:26.588253+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":568,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-30T13:13:02.325684+0000","last_change":"2024-03-30T13:12:23.100937+0000","last_active":"2024-03-30T13:13:02.325684+0000","last_peered":"2024-03-30T13:13:02.325684+0000","last_clean":"2024-03-30T13:13:02.325684+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:13:02.325684+0000","last_undegraded":"2024-03-30T13:13:02.325684+0000","last_fullsized":"2024-03-30T13:13:02.325684+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:12:23.100675+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:12:23.100675+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:44.577727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":387,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-30T13:13:02.323768+0000","last_change":"2024-03-30T13:12:20.783890+0000","last_active":"2024-03-30T13:13:02.323768+0000","last_peered":"2024-03-30T13:13:02.323768+0000","last_clean":"2024-03-30T13:13:02.323768+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:13:02.323768+0000","last_undegraded":"2024-03-30T13:13:02.323768+0000","last_fullsized":"2024-03-30T13:13:02.323768+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:20.783652+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:12:20.783652+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:32.641222+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1054,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-30T13:13:06.936133+0000","last_change":"2024-03-30T13:12:15.741547+0000","last_active":"2024-03-30T13:13:06.936133+0000","last_peered":"2024-03-30T13:13:06.936133+0000","last_clean":"2024-03-30T13:13:06.936133+0000","last_became_active":"2024-03-30T13:10:18.181149+0000","last_became_peered":"2024-03-30T13:10:18.181149+0000","last_unstale":"2024-03-30T13:13:06.936133+0000","last_undegraded":"2024-03-30T13:13:06.936133+0000","last_fullsized":"2024-03-30T13:13:06.936133+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:12:15.741379+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:36.529709+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":603,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-30T13:13:02.325717+0000","last_change":"2024-03-30T13:12:04.242615+0000","last_active":"2024-03-30T13:13:02.325717+0000","last_peered":"2024-03-30T13:13:02.325717+0000","last_clean":"2024-03-30T13:13:02.325717+0000","last_became_active":"2024-03-30T13:12:00.202949+0000","last_became_peered":"2024-03-30T13:12:00.202949+0000","last_unstale":"2024-03-30T13:13:02.325717+0000","last_undegraded":"2024-03-30T13:13:02.325717+0000","last_fullsized":"2024-03-30T13:13:02.325717+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:12:04.242418+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:12:04.242418+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:12.083440+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":321,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-03-30T13:13:04.352206+0000","last_change":"2024-03-30T13:12:51.513374+0000","last_active":"2024-03-30T13:13:04.352206+0000","last_peered":"2024-03-30T13:13:04.352206+0000","last_clean":"2024-03-30T13:13:04.352206+0000","last_became_active":"2024-03-30T13:12:51.512874+0000","last_became_peered":"2024-03-30T13:12:51.512874+0000","last_unstale":"2024-03-30T13:13:04.352206+0000","last_undegraded":"2024-03-30T13:13:04.352206+0000","last_fullsized":"2024-03-30T13:13:04.352206+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:11:54.706131+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:11:54.706131+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:05.329049+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1172,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-30T13:13:06.936144+0000","last_change":"2024-03-30T13:11:54.781544+0000","last_active":"2024-03-30T13:13:06.936144+0000","last_peered":"2024-03-30T13:13:06.936144+0000","last_clean":"2024-03-30T13:13:06.936144+0000","last_became_active":"2024-03-30T13:11:09.212712+0000","last_became_peered":"2024-03-30T13:11:09.212712+0000","last_unstale":"2024-03-30T13:13:06.936144+0000","last_undegraded":"2024-03-30T13:13:06.936144+0000","last_fullsized":"2024-03-30T13:13:06.936144+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:54.781288+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:11:54.781288+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:21.599312+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":55832128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":888,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-30T13:13:03.348682+0000","last_change":"2024-03-30T13:12:05.457652+0000","last_active":"2024-03-30T13:13:03.348682+0000","last_peered":"2024-03-30T13:13:03.348682+0000","last_clean":"2024-03-30T13:13:03.348682+0000","last_became_active":"2024-03-30T13:12:00.203605+0000","last_became_peered":"2024-03-30T13:12:00.203605+0000","last_unstale":"2024-03-30T13:13:03.348682+0000","last_undegraded":"2024-03-30T13:13:03.348682+0000","last_fullsized":"2024-03-30T13:13:03.348682+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:05.457409+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:12:05.457409+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:32.879605+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1080,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-03-30T13:13:05.850760+0000","last_change":"2024-03-30T13:12:26.917834+0000","last_active":"2024-03-30T13:13:05.850760+0000","last_peered":"2024-03-30T13:13:05.850760+0000","last_clean":"2024-03-30T13:13:05.850760+0000","last_became_active":"2024-03-30T13:12:00.202952+0000","last_became_peered":"2024-03-30T13:12:00.202952+0000","last_unstale":"2024-03-30T13:13:05.850760+0000","last_undegraded":"2024-03-30T13:13:05.850760+0000","last_fullsized":"2024-03-30T13:13:05.850760+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:12:26.917553+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:56.679031+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1121,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-03-30T13:13:05.850761+0000","last_change":"2024-03-30T13:12:33.835689+0000","last_active":"2024-03-30T13:13:05.850761+0000","last_peered":"2024-03-30T13:13:05.850761+0000","last_clean":"2024-03-30T13:13:05.850761+0000","last_became_active":"2024-03-30T13:10:18.180670+0000","last_became_peered":"2024-03-30T13:10:18.180670+0000","last_unstale":"2024-03-30T13:13:05.850761+0000","last_undegraded":"2024-03-30T13:13:05.850761+0000","last_fullsized":"2024-03-30T13:13:05.850761+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:12:33.835431+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:12:33.835431+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:49.355370+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":425,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-30T13:13:02.323761+0000","last_change":"2024-03-30T13:12:36.623829+0000","last_active":"2024-03-30T13:13:02.323761+0000","last_peered":"2024-03-30T13:13:02.323761+0000","last_clean":"2024-03-30T13:13:02.323761+0000","last_became_active":"2024-03-30T13:10:18.183814+0000","last_became_peered":"2024-03-30T13:10:18.183814+0000","last_unstale":"2024-03-30T13:13:02.323761+0000","last_undegraded":"2024-03-30T13:13:02.323761+0000","last_fullsized":"2024-03-30T13:13:02.323761+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:12:36.623574+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:12:36.623574+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:58.642559+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":539,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-30T13:13:02.325684+0000","last_change":"2024-03-30T13:12:18.090983+0000","last_active":"2024-03-30T13:13:02.325684+0000","last_peered":"2024-03-30T13:13:02.325684+0000","last_clean":"2024-03-30T13:13:02.325684+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:13:02.325684+0000","last_undegraded":"2024-03-30T13:13:02.325684+0000","last_fullsized":"2024-03-30T13:13:02.325684+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:18.090642+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:12:18.090642+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:27.145345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":425,"reported_epoch":473,"state":"active+clean","last_fresh":"2024-03-30T13:13:02.324033+0000","last_change":"2024-03-30T13:11:55.853801+0000","last_active":"2024-03-30T13:13:02.324033+0000","last_peered":"2024-03-30T13:13:02.324033+0000","last_clean":"2024-03-30T13:13:02.324033+0000","last_became_active":"2024-03-30T13:09:27.162655+0000","last_became_peered":"2024-03-30T13:09:27.162655+0000","last_unstale":"2024-03-30T13:13:02.324033+0000","last_undegraded":"2024-03-30T13:13:02.324033+0000","last_fullsized":"2024-03-30T13:13:02.324033+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:11:55.853519+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:11:55.853519+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:05.040598+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":278,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-03-30T13:13:04.352217+0000","last_change":"2024-03-30T13:12:51.515035+0000","last_active":"2024-03-30T13:13:04.352217+0000","last_peered":"2024-03-30T13:13:04.352217+0000","last_clean":"2024-03-30T13:13:04.352217+0000","last_became_active":"2024-03-30T13:12:51.513803+0000","last_became_peered":"2024-03-30T13:12:51.513803+0000","last_unstale":"2024-03-30T13:13:04.352217+0000","last_undegraded":"2024-03-30T13:13:04.352217+0000","last_fullsized":"2024-03-30T13:13:04.352217+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:12:45.634758+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:12:45.634758+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:05.867044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":226992306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834158,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52900,"kb_used_data":20760,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94318940,"statfs":{"total":96636764160,"available":96582594560,"internally_reserved":0,"allocated":21258240,"data_stored":18074518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":320,"seq":1374389534786,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62232,"kb_used_data":25480,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94309608,"statfs":{"total":96636764160,"available":96573038592,"internally_reserved":0,"allocated":26091520,"data_stored":22921400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7704,"internal_metadata":37609960},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":441,"seq":1894080577551,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54400,"kb_used_data":17196,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94317440,"statfs":{"total":96636764160,"available":96581058560,"internally_reserved":0,"allocated":17608704,"data_stored":14431783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9343,"internal_metadata":38067073},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509654,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47300,"kb_used_data":11952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94324540,"statfs":{"total":96636764160,"available":96588328960,"internally_reserved":0,"allocated":12238848,"data_stored":9068728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":461,"seq":1979979923460,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43840,"kb_used_data":11252,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94328000,"statfs":{"total":96636764160,"available":96591872000,"internally_reserved":0,"allocated":11522048,"data_stored":8338326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7692,"internal_metadata":33350132},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509655,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50316,"kb_used_data":19128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321524,"statfs":{"total":96636764160,"available":96585240576,"internally_reserved":0,"allocated":19587072,"data_stored":16416952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297049,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53320,"kb_used_data":15672,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94318520,"statfs":{"total":96636764160,"available":96582164480,"internally_reserved":0,"allocated":16048128,"data_stored":12878008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147364,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49860,"kb_used_data":18164,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321980,"statfs":{"total":96636764160,"available":96585707520,"internally_reserved":0,"allocated":18599936,"data_stored":15409413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:13:08.308 INFO:tasks.thrashosds.thrasher:Choosing PG 1.8 with acting set [5, 2, 3] 2024-03-30T13:13:08.308 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 2, 5, 6, 3, 7, 4, 1] 2024-03-30T13:13:08.308 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-30T13:13:08.308 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:13:08.379 INFO:tasks.ceph.osd.5:Stopped 2024-03-30T13:13:08.380 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 2, 5, 6, 3, 7, 4, 1] 2024-03-30T13:13:08.380 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 5 2024-03-30T13:13:08.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:08.489+0000 7f64392f4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:08.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:08.505+0000 7f64392f4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:08.680 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:13:08.677+0000 7fbb38f08640 -1 mon.a@0(leader).osd e479 definitely_dead 0 2024-03-30T13:13:09.399 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.5. 2024-03-30T13:13:09.434 INFO:tasks.thrashosds.thrasher:dead_osds=[5], live_osds=[0, 2, 6, 3, 7, 4, 1] 2024-03-30T13:13:09.434 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:13:09.434 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:13:09.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:09.553+0000 7ff82a33d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:09.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:09.565+0000 7ff82a33d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:09.740 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:13:09.741 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:13:09.759 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1001,"stamp":"2024-03-30T13:13:09.403352+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":226992306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":352052,"kb_used_data":114244,"kb_used_omap":42,"kb_used_meta":237653,"kb_avail":660250828,"statfs":{"total":676457349120,"available":676096847872,"internally_reserved":0,"allocated":116985856,"data_stored":94672921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43307,"internal_metadata":243357397},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.038145"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1067,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-03-30T13:13:05.850815+0000","last_change":"2024-03-30T13:12:53.896798+0000","last_active":"2024-03-30T13:13:05.850815+0000","last_peered":"2024-03-30T13:13:05.850815+0000","last_clean":"2024-03-30T13:13:05.850815+0000","last_became_active":"2024-03-30T13:12:00.205621+0000","last_became_peered":"2024-03-30T13:12:00.205621+0000","last_unstale":"2024-03-30T13:13:05.850815+0000","last_undegraded":"2024-03-30T13:13:05.850815+0000","last_fullsized":"2024-03-30T13:13:05.850815+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:12:53.896535+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:12:53.896535+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:04.066529+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1024,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-30T13:13:08.402783+0000","last_change":"2024-03-30T13:12:09.454216+0000","last_active":"2024-03-30T13:13:08.402783+0000","last_peered":"2024-03-30T13:13:08.402783+0000","last_clean":"2024-03-30T13:13:08.402783+0000","last_became_active":"2024-03-30T13:12:00.216256+0000","last_became_peered":"2024-03-30T13:12:00.216256+0000","last_unstale":"2024-03-30T13:13:08.402783+0000","last_undegraded":"2024-03-30T13:13:08.402783+0000","last_fullsized":"2024-03-30T13:13:08.402783+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:09.453958+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_clean_scrub_stamp":"2024-03-30T13:12:09.453958+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:33.947990+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":636,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-30T13:13:07.637938+0000","last_change":"2024-03-30T13:12:51.213469+0000","last_active":"2024-03-30T13:13:07.637938+0000","last_peered":"2024-03-30T13:13:07.637938+0000","last_clean":"2024-03-30T13:13:07.637938+0000","last_became_active":"2024-03-30T13:12:51.212130+0000","last_became_peered":"2024-03-30T13:12:51.212130+0000","last_unstale":"2024-03-30T13:13:07.637938+0000","last_undegraded":"2024-03-30T13:13:07.637938+0000","last_fullsized":"2024-03-30T13:13:07.637938+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:12:00.319122+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:12:00.319122+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:26.588253+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":579,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-30T13:13:07.637439+0000","last_change":"2024-03-30T13:12:23.100937+0000","last_active":"2024-03-30T13:13:07.637439+0000","last_peered":"2024-03-30T13:13:07.637439+0000","last_clean":"2024-03-30T13:13:07.637439+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:13:07.637439+0000","last_undegraded":"2024-03-30T13:13:07.637439+0000","last_fullsized":"2024-03-30T13:13:07.637439+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:12:23.100675+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:12:23.100675+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:44.577727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":387,"reported_epoch":473,"state":"stale+active+clean","last_fresh":"2024-03-30T13:13:02.323768+0000","last_change":"2024-03-30T13:12:20.783890+0000","last_active":"2024-03-30T13:13:02.323768+0000","last_peered":"2024-03-30T13:13:02.323768+0000","last_clean":"2024-03-30T13:13:02.323768+0000","last_became_active":"2024-03-30T13:07:45.155146+0000","last_became_peered":"2024-03-30T13:07:45.155146+0000","last_unstale":"2024-03-30T13:13:08.400575+0000","last_undegraded":"2024-03-30T13:13:02.323768+0000","last_fullsized":"2024-03-30T13:13:02.323768+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:20.783652+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:12:20.783652+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:32.641222+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1054,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-30T13:13:06.936133+0000","last_change":"2024-03-30T13:12:15.741547+0000","last_active":"2024-03-30T13:13:06.936133+0000","last_peered":"2024-03-30T13:13:06.936133+0000","last_clean":"2024-03-30T13:13:06.936133+0000","last_became_active":"2024-03-30T13:10:18.181149+0000","last_became_peered":"2024-03-30T13:10:18.181149+0000","last_unstale":"2024-03-30T13:13:06.936133+0000","last_undegraded":"2024-03-30T13:13:06.936133+0000","last_fullsized":"2024-03-30T13:13:06.936133+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:12:15.741379+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:36.529709+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":613,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-30T13:13:07.637436+0000","last_change":"2024-03-30T13:12:04.242615+0000","last_active":"2024-03-30T13:13:07.637436+0000","last_peered":"2024-03-30T13:13:07.637436+0000","last_clean":"2024-03-30T13:13:07.637436+0000","last_became_active":"2024-03-30T13:12:00.202949+0000","last_became_peered":"2024-03-30T13:12:00.202949+0000","last_unstale":"2024-03-30T13:13:07.637436+0000","last_undegraded":"2024-03-30T13:13:07.637436+0000","last_fullsized":"2024-03-30T13:13:07.637436+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:12:04.242418+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:12:04.242418+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:12.083440+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":321,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-03-30T13:13:04.352206+0000","last_change":"2024-03-30T13:12:51.513374+0000","last_active":"2024-03-30T13:13:04.352206+0000","last_peered":"2024-03-30T13:13:04.352206+0000","last_clean":"2024-03-30T13:13:04.352206+0000","last_became_active":"2024-03-30T13:12:51.512874+0000","last_became_peered":"2024-03-30T13:12:51.512874+0000","last_unstale":"2024-03-30T13:13:04.352206+0000","last_undegraded":"2024-03-30T13:13:04.352206+0000","last_fullsized":"2024-03-30T13:13:04.352206+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:11:54.706131+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:11:54.706131+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:05.329049+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":44067360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1172,"reported_epoch":477,"state":"active+clean","last_fresh":"2024-03-30T13:13:06.936144+0000","last_change":"2024-03-30T13:11:54.781544+0000","last_active":"2024-03-30T13:13:06.936144+0000","last_peered":"2024-03-30T13:13:06.936144+0000","last_clean":"2024-03-30T13:13:06.936144+0000","last_became_active":"2024-03-30T13:11:09.212712+0000","last_became_peered":"2024-03-30T13:11:09.212712+0000","last_unstale":"2024-03-30T13:13:06.936144+0000","last_undegraded":"2024-03-30T13:13:06.936144+0000","last_fullsized":"2024-03-30T13:13:06.936144+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:11:54.781288+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:11:54.781288+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:21.599312+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":55832128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":898,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-03-30T13:13:08.402246+0000","last_change":"2024-03-30T13:12:05.457652+0000","last_active":"2024-03-30T13:13:08.402246+0000","last_peered":"2024-03-30T13:13:08.402246+0000","last_clean":"2024-03-30T13:13:08.402246+0000","last_became_active":"2024-03-30T13:12:00.203605+0000","last_became_peered":"2024-03-30T13:12:00.203605+0000","last_unstale":"2024-03-30T13:13:08.402246+0000","last_undegraded":"2024-03-30T13:13:08.402246+0000","last_fullsized":"2024-03-30T13:13:08.402246+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:05.457409+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:12:05.457409+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:32.879605+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1080,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-03-30T13:13:05.850760+0000","last_change":"2024-03-30T13:12:26.917834+0000","last_active":"2024-03-30T13:13:05.850760+0000","last_peered":"2024-03-30T13:13:05.850760+0000","last_clean":"2024-03-30T13:13:05.850760+0000","last_became_active":"2024-03-30T13:12:00.202952+0000","last_became_peered":"2024-03-30T13:12:00.202952+0000","last_unstale":"2024-03-30T13:13:05.850760+0000","last_undegraded":"2024-03-30T13:13:05.850760+0000","last_fullsized":"2024-03-30T13:13:05.850760+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:12:26.917553+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:56.679031+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1121,"reported_epoch":476,"state":"active+clean","last_fresh":"2024-03-30T13:13:05.850761+0000","last_change":"2024-03-30T13:12:33.835689+0000","last_active":"2024-03-30T13:13:05.850761+0000","last_peered":"2024-03-30T13:13:05.850761+0000","last_clean":"2024-03-30T13:13:05.850761+0000","last_became_active":"2024-03-30T13:10:18.180670+0000","last_became_peered":"2024-03-30T13:10:18.180670+0000","last_unstale":"2024-03-30T13:13:05.850761+0000","last_undegraded":"2024-03-30T13:13:05.850761+0000","last_fullsized":"2024-03-30T13:13:05.850761+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:12:33.835431+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:12:33.835431+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:49.355370+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":425,"reported_epoch":473,"state":"stale+active+clean","last_fresh":"2024-03-30T13:13:02.323761+0000","last_change":"2024-03-30T13:12:36.623829+0000","last_active":"2024-03-30T13:13:02.323761+0000","last_peered":"2024-03-30T13:13:02.323761+0000","last_clean":"2024-03-30T13:13:02.323761+0000","last_became_active":"2024-03-30T13:10:18.183814+0000","last_became_peered":"2024-03-30T13:10:18.183814+0000","last_unstale":"2024-03-30T13:13:08.400579+0000","last_undegraded":"2024-03-30T13:13:02.323761+0000","last_fullsized":"2024-03-30T13:13:02.323761+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:12:36.623574+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:12:36.623574+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:58.642559+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":549,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-03-30T13:13:07.637284+0000","last_change":"2024-03-30T13:12:18.090983+0000","last_active":"2024-03-30T13:13:07.637284+0000","last_peered":"2024-03-30T13:13:07.637284+0000","last_clean":"2024-03-30T13:13:07.637284+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:13:07.637284+0000","last_undegraded":"2024-03-30T13:13:07.637284+0000","last_fullsized":"2024-03-30T13:13:07.637284+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:18.090642+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:12:18.090642+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:27.145345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":425,"reported_epoch":473,"state":"stale+active+clean","last_fresh":"2024-03-30T13:13:02.324033+0000","last_change":"2024-03-30T13:11:55.853801+0000","last_active":"2024-03-30T13:13:02.324033+0000","last_peered":"2024-03-30T13:13:02.324033+0000","last_clean":"2024-03-30T13:13:02.324033+0000","last_became_active":"2024-03-30T13:09:27.162655+0000","last_became_peered":"2024-03-30T13:09:27.162655+0000","last_unstale":"2024-03-30T13:13:08.400581+0000","last_undegraded":"2024-03-30T13:13:02.324033+0000","last_fullsized":"2024-03-30T13:13:02.324033+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:11:55.853519+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:11:55.853519+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:05.040598+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":278,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-03-30T13:13:04.352217+0000","last_change":"2024-03-30T13:12:51.515035+0000","last_active":"2024-03-30T13:13:04.352217+0000","last_peered":"2024-03-30T13:13:04.352217+0000","last_clean":"2024-03-30T13:13:04.352217+0000","last_became_active":"2024-03-30T13:12:51.513803+0000","last_became_peered":"2024-03-30T13:12:51.513803+0000","last_unstale":"2024-03-30T13:13:04.352217+0000","last_undegraded":"2024-03-30T13:13:04.352217+0000","last_fullsized":"2024-03-30T13:13:04.352217+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:12:45.634758+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:12:45.634758+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:05.867044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":226992306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834158,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52900,"kb_used_data":20760,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94318940,"statfs":{"total":96636764160,"available":96582594560,"internally_reserved":0,"allocated":21258240,"data_stored":18074518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":441,"seq":1894080577552,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54456,"kb_used_data":17256,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94317384,"statfs":{"total":96636764160,"available":96581001216,"internally_reserved":0,"allocated":17670144,"data_stored":14459499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9343,"internal_metadata":38067073},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509654,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47300,"kb_used_data":11952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94324540,"statfs":{"total":96636764160,"available":96588328960,"internally_reserved":0,"allocated":12238848,"data_stored":9068728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":461,"seq":1979979923460,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43840,"kb_used_data":11252,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94328000,"statfs":{"total":96636764160,"available":96591872000,"internally_reserved":0,"allocated":11522048,"data_stored":8338326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7692,"internal_metadata":33350132},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509655,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50316,"kb_used_data":19128,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321524,"statfs":{"total":96636764160,"available":96585240576,"internally_reserved":0,"allocated":19587072,"data_stored":16416952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297050,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53380,"kb_used_data":15732,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94318460,"statfs":{"total":96636764160,"available":96582103040,"internally_reserved":0,"allocated":16109568,"data_stored":12905485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147364,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49860,"kb_used_data":18164,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321980,"statfs":{"total":96636764160,"available":96585707520,"internally_reserved":0,"allocated":18599936,"data_stored":15409413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:13:09.760 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:13:34.761 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:13:34.761 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:13:34.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:34.888+0000 7f0eb9a3d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:34.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:34.900+0000 7f0eb9a3d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:35.080 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:13:35.080 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:13:35.098 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1020,"stamp":"2024-03-30T13:13:34.094268+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":282101700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":372748,"kb_used_data":134972,"kb_used_omap":42,"kb_used_meta":237653,"kb_avail":660230132,"statfs":{"total":676457349120,"available":676075655168,"internally_reserved":0,"allocated":138211328,"data_stored":115405776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43307,"internal_metadata":243357397},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002972"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1098,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-30T13:13:25.686755+0000","last_change":"2024-03-30T13:13:10.482891+0000","last_active":"2024-03-30T13:13:25.686755+0000","last_peered":"2024-03-30T13:13:25.686755+0000","last_clean":"2024-03-30T13:13:25.686755+0000","last_became_active":"2024-03-30T13:13:10.427181+0000","last_became_peered":"2024-03-30T13:13:10.427181+0000","last_unstale":"2024-03-30T13:13:25.686755+0000","last_undegraded":"2024-03-30T13:13:25.686755+0000","last_fullsized":"2024-03-30T13:13:25.686755+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:12:53.896535+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:12:53.896535+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:14.268080+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":13764735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1038,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-30T13:13:25.178905+0000","last_change":"2024-03-30T13:12:09.454216+0000","last_active":"2024-03-30T13:13:25.178905+0000","last_peered":"2024-03-30T13:13:25.178905+0000","last_clean":"2024-03-30T13:13:25.178905+0000","last_became_active":"2024-03-30T13:12:00.216256+0000","last_became_peered":"2024-03-30T13:12:00.216256+0000","last_unstale":"2024-03-30T13:13:25.178905+0000","last_undegraded":"2024-03-30T13:13:25.178905+0000","last_fullsized":"2024-03-30T13:13:25.178905+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:09.453958+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_clean_scrub_stamp":"2024-03-30T13:12:09.453958+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:33.947990+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":661,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-30T13:13:28.024903+0000","last_change":"2024-03-30T13:13:28.025038+0000","last_active":"2024-03-30T13:13:28.024903+0000","last_peered":"2024-03-30T13:13:28.024903+0000","last_clean":"2024-03-30T13:13:28.024903+0000","last_became_active":"2024-03-30T13:12:51.212130+0000","last_became_peered":"2024-03-30T13:12:51.212130+0000","last_unstale":"2024-03-30T13:13:28.024903+0000","last_undegraded":"2024-03-30T13:13:28.024903+0000","last_fullsized":"2024-03-30T13:13:28.024903+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:13:28.024596+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:13:28.024596+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:47.470109+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":595,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-30T13:13:25.176393+0000","last_change":"2024-03-30T13:12:23.100937+0000","last_active":"2024-03-30T13:13:25.176393+0000","last_peered":"2024-03-30T13:13:25.176393+0000","last_clean":"2024-03-30T13:13:25.176393+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:13:25.176393+0000","last_undegraded":"2024-03-30T13:13:25.176393+0000","last_fullsized":"2024-03-30T13:13:25.176393+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:12:23.100675+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:12:23.100675+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:44.577727+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":25,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-30T13:13:25.178645+0000","last_change":"2024-03-30T13:13:10.511522+0000","last_active":"2024-03-30T13:13:25.178645+0000","last_peered":"2024-03-30T13:13:25.178645+0000","last_clean":"2024-03-30T13:13:25.178645+0000","last_became_active":"2024-03-30T13:13:10.424195+0000","last_became_peered":"2024-03-30T13:13:10.424195+0000","last_unstale":"2024-03-30T13:13:25.178645+0000","last_undegraded":"2024-03-30T13:13:25.178645+0000","last_fullsized":"2024-03-30T13:13:25.178645+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:20.783652+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:12:20.783652+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:47.229916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1089,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-30T13:13:25.509146+0000","last_change":"2024-03-30T13:13:10.615405+0000","last_active":"2024-03-30T13:13:25.509146+0000","last_peered":"2024-03-30T13:13:25.509146+0000","last_clean":"2024-03-30T13:13:25.509146+0000","last_became_active":"2024-03-30T13:13:10.425217+0000","last_became_peered":"2024-03-30T13:13:10.425217+0000","last_unstale":"2024-03-30T13:13:25.509146+0000","last_undegraded":"2024-03-30T13:13:25.509146+0000","last_fullsized":"2024-03-30T13:13:25.509146+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:12:15.741379+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:34.659281+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11394981,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":638,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-30T13:13:25.176490+0000","last_change":"2024-03-30T13:13:16.070252+0000","last_active":"2024-03-30T13:13:25.176490+0000","last_peered":"2024-03-30T13:13:25.176490+0000","last_clean":"2024-03-30T13:13:25.176490+0000","last_became_active":"2024-03-30T13:12:00.202949+0000","last_became_peered":"2024-03-30T13:12:00.202949+0000","last_unstale":"2024-03-30T13:13:25.176490+0000","last_undegraded":"2024-03-30T13:13:25.176490+0000","last_fullsized":"2024-03-30T13:13:25.176490+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:13:16.069976+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:13:16.069976+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:25.894866+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":364,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-30T13:13:25.172756+0000","last_change":"2024-03-30T13:13:15.593837+0000","last_active":"2024-03-30T13:13:25.172756+0000","last_peered":"2024-03-30T13:13:25.172756+0000","last_clean":"2024-03-30T13:13:25.172756+0000","last_became_active":"2024-03-30T13:13:10.425190+0000","last_became_peered":"2024-03-30T13:13:10.425190+0000","last_unstale":"2024-03-30T13:13:25.172756+0000","last_undegraded":"2024-03-30T13:13:25.172756+0000","last_fullsized":"2024-03-30T13:13:25.172756+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:13:15.593589+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:13:15.593589+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:39.099018+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":51411920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1209,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-30T13:13:25.509038+0000","last_change":"2024-03-30T13:13:12.880017+0000","last_active":"2024-03-30T13:13:25.509038+0000","last_peered":"2024-03-30T13:13:25.509038+0000","last_clean":"2024-03-30T13:13:25.509038+0000","last_became_active":"2024-03-30T13:13:10.429866+0000","last_became_peered":"2024-03-30T13:13:10.429866+0000","last_unstale":"2024-03-30T13:13:25.509038+0000","last_undegraded":"2024-03-30T13:13:25.509038+0000","last_fullsized":"2024-03-30T13:13:25.509038+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:13:12.879768+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:13:12.879768+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:16.566505+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":912,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-30T13:13:25.178760+0000","last_change":"2024-03-30T13:12:05.457652+0000","last_active":"2024-03-30T13:13:25.178760+0000","last_peered":"2024-03-30T13:13:25.178760+0000","last_clean":"2024-03-30T13:13:25.178760+0000","last_became_active":"2024-03-30T13:12:00.203605+0000","last_became_peered":"2024-03-30T13:12:00.203605+0000","last_unstale":"2024-03-30T13:13:25.178760+0000","last_undegraded":"2024-03-30T13:13:25.178760+0000","last_fullsized":"2024-03-30T13:13:25.178760+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:12:05.457409+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:12:05.457409+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:32.879605+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1100,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-30T13:13:25.686713+0000","last_change":"2024-03-30T13:12:26.917834+0000","last_active":"2024-03-30T13:13:25.686713+0000","last_peered":"2024-03-30T13:13:25.686713+0000","last_clean":"2024-03-30T13:13:25.686713+0000","last_became_active":"2024-03-30T13:12:00.202952+0000","last_became_peered":"2024-03-30T13:12:00.202952+0000","last_unstale":"2024-03-30T13:13:25.686713+0000","last_undegraded":"2024-03-30T13:13:25.686713+0000","last_fullsized":"2024-03-30T13:13:25.686713+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:12:26.917553+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:56.679031+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1164,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-30T13:13:25.686690+0000","last_change":"2024-03-30T13:13:11.099133+0000","last_active":"2024-03-30T13:13:25.686690+0000","last_peered":"2024-03-30T13:13:25.686690+0000","last_clean":"2024-03-30T13:13:25.686690+0000","last_became_active":"2024-03-30T13:13:10.423662+0000","last_became_peered":"2024-03-30T13:13:10.423662+0000","last_unstale":"2024-03-30T13:13:25.686690+0000","last_undegraded":"2024-03-30T13:13:25.686690+0000","last_fullsized":"2024-03-30T13:13:25.686690+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:12:33.835431+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:12:33.835431+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:00.362536+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":31,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-30T13:13:25.686830+0000","last_change":"2024-03-30T13:13:10.631757+0000","last_active":"2024-03-30T13:13:25.686830+0000","last_peered":"2024-03-30T13:13:25.686830+0000","last_clean":"2024-03-30T13:13:25.686830+0000","last_became_active":"2024-03-30T13:13:10.427647+0000","last_became_peered":"2024-03-30T13:13:10.427647+0000","last_unstale":"2024-03-30T13:13:25.686830+0000","last_undegraded":"2024-03-30T13:13:25.686830+0000","last_fullsized":"2024-03-30T13:13:25.686830+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:12:36.623574+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:12:36.623574+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:13:56.832006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":574,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-30T13:13:29.018208+0000","last_change":"2024-03-30T13:13:29.018285+0000","last_active":"2024-03-30T13:13:29.018208+0000","last_peered":"2024-03-30T13:13:29.018208+0000","last_clean":"2024-03-30T13:13:29.018208+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:13:29.018208+0000","last_undegraded":"2024-03-30T13:13:29.018208+0000","last_fullsized":"2024-03-30T13:13:29.018208+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:29.018031+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:13:29.018031+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:47.121728+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":31,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-30T13:13:25.178871+0000","last_change":"2024-03-30T13:13:10.516531+0000","last_active":"2024-03-30T13:13:25.178871+0000","last_peered":"2024-03-30T13:13:25.178871+0000","last_clean":"2024-03-30T13:13:25.178871+0000","last_became_active":"2024-03-30T13:13:10.424144+0000","last_became_peered":"2024-03-30T13:13:10.424144+0000","last_unstale":"2024-03-30T13:13:25.178871+0000","last_undegraded":"2024-03-30T13:13:25.178871+0000","last_fullsized":"2024-03-30T13:13:25.178871+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:13:05.217766+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:13:05.217766+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:16.565414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":311,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-03-30T13:13:25.172775+0000","last_change":"2024-03-30T13:13:10.875970+0000","last_active":"2024-03-30T13:13:25.172775+0000","last_peered":"2024-03-30T13:13:25.172775+0000","last_clean":"2024-03-30T13:13:25.172775+0000","last_became_active":"2024-03-30T13:13:10.828594+0000","last_became_peered":"2024-03-30T13:13:10.828594+0000","last_unstale":"2024-03-30T13:13:25.172775+0000","last_undegraded":"2024-03-30T13:13:25.172775+0000","last_fullsized":"2024-03-30T13:13:25.172775+0000","mapping_epoch":480,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:12:45.634758+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:12:45.634758+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:14.513120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5226911,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":282101700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":117080064,"data_stored":117080064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834163,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58596,"kb_used_data":26456,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94313244,"statfs":{"total":96636764160,"available":96576761856,"internally_reserved":0,"allocated":27090944,"data_stored":23833008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":441,"seq":1894080577556,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59032,"kb_used_data":21836,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94312808,"statfs":{"total":96636764160,"available":96576315392,"internally_reserved":0,"allocated":22360064,"data_stored":19102128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9343,"internal_metadata":38067073},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509659,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50076,"kb_used_data":14736,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94321764,"statfs":{"total":96636764160,"available":96585486336,"internally_reserved":0,"allocated":15089664,"data_stored":11831728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":461,"seq":1979979923465,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49184,"kb_used_data":16596,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94322656,"statfs":{"total":96636764160,"available":96586399744,"internally_reserved":0,"allocated":16994304,"data_stored":13736368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7692,"internal_metadata":33350132},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509660,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50464,"kb_used_data":19284,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321376,"statfs":{"total":96636764160,"available":96585089024,"internally_reserved":0,"allocated":19746816,"data_stored":16488880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297055,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53468,"kb_used_data":15828,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94318372,"statfs":{"total":96636764160,"available":96582012928,"internally_reserved":0,"allocated":16207872,"data_stored":12949936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147369,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51928,"kb_used_data":20236,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94319912,"statfs":{"total":96636764160,"available":96583589888,"internally_reserved":0,"allocated":20721664,"data_stored":17463728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11071488,"data_stored":11071488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9166848,"data_stored":9166848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":14798848,"data_stored":14798848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16437248,"data_stored":16437248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":21168128,"data_stored":21168128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:13:35.099 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:13:35.099 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:13:35.099 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-03-30T13:13:35.100 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-30T13:13:35.100 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-30T13:13:35.102 INFO:tasks.ceph.osd.5:Started 2024-03-30T13:13:35.103 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-30T13:13:35.167 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:13:35.166+0000 7ff8539237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:35.174 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:13:35.174+0000 7ff8539237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:35.174 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:13:35.174+0000 7ff8539237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:35.223 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T13:13:35.223 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T13:13:35.224 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T13:13:35.224 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T13:13:35.224 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T13:13:35.224 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T13:13:35.224 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-03-30T13:13:35.224 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-03-30T13:13:35.224 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-03-30T13:13:35.224 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-03-30T13:13:35.224 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-03-30T13:13:35.224 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T13:13:35.226 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:13:35.226 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-03-30T13:13:35.709 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:13:35.710+0000 7ff8539237c0 -1 Falling back to public interface 2024-03-30T13:13:39.498 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:13:39.498+0000 7ff8539237c0 -1 osd.5 478 log_to_monitors true 2024-03-30T13:13:40.227 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-30T13:13:40.352 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T13:13:40.352 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T13:13:40.352 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T13:13:40.352 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T13:13:40.352 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T13:13:40.352 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T13:13:40.352 INFO:teuthology.orchestra.run.smithi082.stderr:bluefs debug_inject_read_zeros 2024-03-30T13:13:40.352 INFO:teuthology.orchestra.run.smithi082.stderr:bluefs files list 2024-03-30T13:13:40.352 INFO:teuthology.orchestra.run.smithi082.stderr:bluefs stats 2024-03-30T13:13:40.352 INFO:teuthology.orchestra.run.smithi082.stderr:bluestore allocator dump block 2024-03-30T13:13:40.353 INFO:teuthology.orchestra.run.smithi082.stderr:bluestore allocator fragmentation block 2024-03-30T13:13:40.353 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T13:13:40.354 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:13:40.355 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-03-30T13:13:40.448 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:13:40.450+0000 7ff84b6f8640 -1 osd.5 478 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:13:45.356 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-30T13:13:45.489 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T13:13:45.490 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T13:13:45.490 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T13:13:45.490 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T13:13:45.504 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:13:45.504 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-03-30T13:13:45.504 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 5 2024-03-30T13:13:45.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:45.628+0000 7f3089e25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:45.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:45.644+0000 7f3089e25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:45.826 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:13:45.824+0000 7fbb38f08640 -1 mon.a@0(leader).osd e489 definitely_dead 0 2024-03-30T13:13:46.460 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.5. 2024-03-30T13:13:46.495 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-03-30T13:13:46.495 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:13:46.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:46.612+0000 7f9220295640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:46.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:46.628+0000 7f9220295640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:47.470 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:13:47.506 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:13:47.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:47.620+0000 7ff884608640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:47.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:47.636+0000 7ff884608640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:48.499 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:13:48.525 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:13:48.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:48.640+0000 7f73cfadb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:48.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:48.656+0000 7f73cfadb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:49.500 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:13:49.530 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:13:49.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:49.648+0000 7f39d56bf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:49.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:49.664+0000 7f39d56bf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:50.497 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:13:50.523 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:13:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:50.640+0000 7fd1d1169640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:50.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:50.656+0000 7fd1d1169640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:51.507 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:13:51.534 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:13:51.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:51.652+0000 7f8582f45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:51.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:51.672+0000 7f8582f45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:52.518 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:13:52.544 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:13:52.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:52.652+0000 7fa0effc2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:52.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:52.672+0000 7fa0effc2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:53.538 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:13:53.562 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:13:53.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:53.684+0000 7f5dafdb8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:53.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:53.700+0000 7f5dafdb8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:54.549 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:13:54.575 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:13:54.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:54.696+0000 7f904b918640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:54.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:54.712+0000 7f904b918640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:55.560 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:13:55.586 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:13:55.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:55.704+0000 7f7dd5f73640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:55.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:55.720+0000 7f7dd5f73640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:56.564 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:13:56.592 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:13:56.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:56.712+0000 7f406f3a0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:56.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:56.728+0000 7f406f3a0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:57.580 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:13:57.604 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:13:57.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:57.720+0000 7f20efa72640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:57.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:57.736+0000 7f20efa72640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:58.591 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:13:58.616 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:13:58.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:58.735+0000 7f6ce330f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:58.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:58.751+0000 7f6ce330f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:59.610 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:13:59.636 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:13:59.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:59.747+0000 7f7047769640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:13:59.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:13:59.763+0000 7f7047769640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:00.616 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:14:00.643 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:14:00.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:00.759+0000 7f1df0d3f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:00.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:00.775+0000 7f1df0d3f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:01.638 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:14:01.663 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:14:01.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:01.779+0000 7fc3c52ec640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:01.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:01.795+0000 7fc3c52ec640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:02.643 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:14:02.669 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:14:02.670 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:14:02.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:02.783+0000 7fb7f20d5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:02.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:02.795+0000 7fb7f20d5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:02.972 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:14:02.973 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:14:02.990 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1054,"stamp":"2024-03-30T13:14:02.647157+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":334975796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":417336,"kb_used_data":142328,"kb_used_omap":51,"kb_used_meta":274828,"kb_avail":754557384,"statfs":{"total":773094113280,"available":772666761216,"internally_reserved":0,"allocated":145743872,"data_stored":118803165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52659,"internal_metadata":281424461},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.051882"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1145,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:00.658623+0000","last_change":"2024-03-30T13:13:47.573184+0000","last_active":"2024-03-30T13:14:00.658623+0000","last_peered":"2024-03-30T13:14:00.658623+0000","last_clean":"2024-03-30T13:14:00.658623+0000","last_became_active":"2024-03-30T13:13:47.489523+0000","last_became_peered":"2024-03-30T13:13:47.489523+0000","last_unstale":"2024-03-30T13:14:00.658623+0000","last_undegraded":"2024-03-30T13:14:00.658623+0000","last_fullsized":"2024-03-30T13:14:00.658623+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:12:53.896535+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:12:53.896535+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:16.650956+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18352980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1078,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-03-30T13:13:58.597583+0000","last_change":"2024-03-30T13:13:42.391739+0000","last_active":"2024-03-30T13:13:58.597583+0000","last_peered":"2024-03-30T13:13:58.597583+0000","last_clean":"2024-03-30T13:13:58.597583+0000","last_became_active":"2024-03-30T13:12:00.216256+0000","last_became_peered":"2024-03-30T13:12:00.216256+0000","last_unstale":"2024-03-30T13:13:58.597583+0000","last_undegraded":"2024-03-30T13:13:58.597583+0000","last_fullsized":"2024-03-30T13:13:58.597583+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:42.391496+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_clean_scrub_stamp":"2024-03-30T13:13:42.391496+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:48.275941+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":689,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-03-30T13:13:57.582197+0000","last_change":"2024-03-30T13:13:28.025038+0000","last_active":"2024-03-30T13:13:57.582197+0000","last_peered":"2024-03-30T13:13:57.582197+0000","last_clean":"2024-03-30T13:13:57.582197+0000","last_became_active":"2024-03-30T13:12:51.212130+0000","last_became_peered":"2024-03-30T13:12:51.212130+0000","last_unstale":"2024-03-30T13:13:57.582197+0000","last_undegraded":"2024-03-30T13:13:57.582197+0000","last_fullsized":"2024-03-30T13:13:57.582197+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:13:28.024596+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:13:28.024596+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:47.470109+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":631,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-03-30T13:13:57.582088+0000","last_change":"2024-03-30T13:13:47.119549+0000","last_active":"2024-03-30T13:13:57.582088+0000","last_peered":"2024-03-30T13:13:57.582088+0000","last_clean":"2024-03-30T13:13:57.582088+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:13:57.582088+0000","last_undegraded":"2024-03-30T13:13:57.582088+0000","last_fullsized":"2024-03-30T13:13:57.582088+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:13:47.119125+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:13:47.119125+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:15.963858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":49,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:01.048663+0000","last_change":"2024-03-30T13:13:49.332620+0000","last_active":"2024-03-30T13:14:01.048663+0000","last_peered":"2024-03-30T13:14:01.048663+0000","last_clean":"2024-03-30T13:14:01.048663+0000","last_became_active":"2024-03-30T13:13:47.479287+0000","last_became_peered":"2024-03-30T13:13:47.479287+0000","last_unstale":"2024-03-30T13:14:01.048663+0000","last_undegraded":"2024-03-30T13:14:01.048663+0000","last_fullsized":"2024-03-30T13:14:01.048663+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:49.332375+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:13:49.332375+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:05.013702+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1151,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:01.046998+0000","last_change":"2024-03-30T13:13:48.841791+0000","last_active":"2024-03-30T13:14:01.046998+0000","last_peered":"2024-03-30T13:14:01.046998+0000","last_clean":"2024-03-30T13:14:01.046998+0000","last_became_active":"2024-03-30T13:13:47.486556+0000","last_became_peered":"2024-03-30T13:13:47.486556+0000","last_unstale":"2024-03-30T13:14:01.046998+0000","last_undegraded":"2024-03-30T13:14:01.046998+0000","last_fullsized":"2024-03-30T13:14:01.046998+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:13:35.802156+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:13:35.802156+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:49.445337+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22789962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":666,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-03-30T13:13:57.582146+0000","last_change":"2024-03-30T13:13:16.070252+0000","last_active":"2024-03-30T13:13:57.582146+0000","last_peered":"2024-03-30T13:13:57.582146+0000","last_clean":"2024-03-30T13:13:57.582146+0000","last_became_active":"2024-03-30T13:12:00.202949+0000","last_became_peered":"2024-03-30T13:12:00.202949+0000","last_unstale":"2024-03-30T13:13:57.582146+0000","last_undegraded":"2024-03-30T13:13:57.582146+0000","last_fullsized":"2024-03-30T13:13:57.582146+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:13:16.069976+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:13:16.069976+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:25.894866+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":412,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-03-30T13:13:59.597660+0000","last_change":"2024-03-30T13:13:48.760543+0000","last_active":"2024-03-30T13:13:59.597660+0000","last_peered":"2024-03-30T13:13:59.597660+0000","last_clean":"2024-03-30T13:13:59.597660+0000","last_became_active":"2024-03-30T13:13:47.499413+0000","last_became_peered":"2024-03-30T13:13:47.499413+0000","last_unstale":"2024-03-30T13:13:59.597660+0000","last_undegraded":"2024-03-30T13:13:59.597660+0000","last_fullsized":"2024-03-30T13:13:59.597660+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:13:15.593589+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:13:15.593589+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:35.673056+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1257,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:01.046711+0000","last_change":"2024-03-30T13:13:47.536108+0000","last_active":"2024-03-30T13:14:01.046711+0000","last_peered":"2024-03-30T13:14:01.046711+0000","last_clean":"2024-03-30T13:14:01.046711+0000","last_became_active":"2024-03-30T13:13:47.486649+0000","last_became_peered":"2024-03-30T13:13:47.486649+0000","last_unstale":"2024-03-30T13:14:01.046711+0000","last_undegraded":"2024-03-30T13:14:01.046711+0000","last_fullsized":"2024-03-30T13:14:01.046711+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:13:12.879768+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:13:12.879768+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:31.285371+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":65983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":952,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-03-30T13:13:58.597508+0000","last_change":"2024-03-30T13:13:38.374659+0000","last_active":"2024-03-30T13:13:58.597508+0000","last_peered":"2024-03-30T13:13:58.597508+0000","last_clean":"2024-03-30T13:13:58.597508+0000","last_became_active":"2024-03-30T13:12:00.203605+0000","last_became_peered":"2024-03-30T13:12:00.203605+0000","last_unstale":"2024-03-30T13:13:58.597508+0000","last_undegraded":"2024-03-30T13:13:58.597508+0000","last_fullsized":"2024-03-30T13:13:58.597508+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:38.374502+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:13:38.374502+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:47.293366+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1143,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:00.658439+0000","last_change":"2024-03-30T13:13:57.067928+0000","last_active":"2024-03-30T13:14:00.658439+0000","last_peered":"2024-03-30T13:14:00.658439+0000","last_clean":"2024-03-30T13:14:00.658439+0000","last_became_active":"2024-03-30T13:12:00.202952+0000","last_became_peered":"2024-03-30T13:12:00.202952+0000","last_unstale":"2024-03-30T13:14:00.658439+0000","last_undegraded":"2024-03-30T13:14:00.658439+0000","last_fullsized":"2024-03-30T13:14:00.658439+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:13:57.067688+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:13:57.067688+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:13.208052+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1231,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:00.658433+0000","last_change":"2024-03-30T13:13:48.914104+0000","last_active":"2024-03-30T13:14:00.658433+0000","last_peered":"2024-03-30T13:14:00.658433+0000","last_clean":"2024-03-30T13:14:00.658433+0000","last_became_active":"2024-03-30T13:13:47.498724+0000","last_became_peered":"2024-03-30T13:13:47.498724+0000","last_unstale":"2024-03-30T13:14:00.658433+0000","last_undegraded":"2024-03-30T13:14:00.658433+0000","last_fullsized":"2024-03-30T13:14:00.658433+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:13:48.913862+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:13:48.913862+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:01.275068+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":57731253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":55,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:01.048630+0000","last_change":"2024-03-30T13:13:52.332055+0000","last_active":"2024-03-30T13:14:01.048630+0000","last_peered":"2024-03-30T13:14:01.048630+0000","last_clean":"2024-03-30T13:14:01.048630+0000","last_became_active":"2024-03-30T13:13:47.483197+0000","last_became_peered":"2024-03-30T13:13:47.483197+0000","last_unstale":"2024-03-30T13:14:01.048630+0000","last_undegraded":"2024-03-30T13:14:01.048630+0000","last_fullsized":"2024-03-30T13:14:01.048630+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:13:52.331764+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:13:52.331764+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:15.424847+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":602,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-03-30T13:13:57.582113+0000","last_change":"2024-03-30T13:13:29.018285+0000","last_active":"2024-03-30T13:13:57.582113+0000","last_peered":"2024-03-30T13:13:57.582113+0000","last_clean":"2024-03-30T13:13:57.582113+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:13:57.582113+0000","last_undegraded":"2024-03-30T13:13:57.582113+0000","last_fullsized":"2024-03-30T13:13:57.582113+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:29.018031+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:13:29.018031+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:47.121728+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":47,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:01.048681+0000","last_change":"2024-03-30T13:13:47.578888+0000","last_active":"2024-03-30T13:14:01.048681+0000","last_peered":"2024-03-30T13:14:01.048681+0000","last_clean":"2024-03-30T13:14:01.048681+0000","last_became_active":"2024-03-30T13:13:47.496794+0000","last_became_peered":"2024-03-30T13:13:47.496794+0000","last_unstale":"2024-03-30T13:14:01.048681+0000","last_undegraded":"2024-03-30T13:14:01.048681+0000","last_fullsized":"2024-03-30T13:14:01.048681+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:13:05.217766+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:13:05.217766+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:07.019567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":357,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-03-30T13:13:59.597762+0000","last_change":"2024-03-30T13:13:47.536417+0000","last_active":"2024-03-30T13:13:59.597762+0000","last_peered":"2024-03-30T13:13:59.597762+0000","last_clean":"2024-03-30T13:13:59.597762+0000","last_became_active":"2024-03-30T13:13:47.481478+0000","last_became_peered":"2024-03-30T13:13:47.481478+0000","last_unstale":"2024-03-30T13:13:59.597762+0000","last_undegraded":"2024-03-30T13:13:59.597762+0000","last_fullsized":"2024-03-30T13:13:59.597762+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:12:45.634758+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:12:45.634758+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:14.539491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10453822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":334975796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834169,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53256,"kb_used_data":21108,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94318584,"statfs":{"total":96636764160,"available":96582230016,"internally_reserved":0,"allocated":21614592,"data_stored":18235952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":488,"seq":2095944040454,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63048,"kb_used_data":25852,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94308792,"statfs":{"total":96636764160,"available":96572203008,"internally_reserved":0,"allocated":26472448,"data_stored":23093808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9352,"internal_metadata":38067064},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":441,"seq":1894080577562,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54724,"kb_used_data":17520,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94317116,"statfs":{"total":96636764160,"available":96580726784,"internally_reserved":0,"allocated":17940480,"data_stored":14582243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9343,"internal_metadata":38067073},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509665,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47648,"kb_used_data":12300,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94324192,"statfs":{"total":96636764160,"available":96587972608,"internally_reserved":0,"allocated":12595200,"data_stored":9230162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":461,"seq":1979979923471,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44164,"kb_used_data":11576,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94327676,"statfs":{"total":96636764160,"available":96591540224,"internally_reserved":0,"allocated":11853824,"data_stored":8488786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7692,"internal_metadata":33350132},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509666,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50644,"kb_used_data":19464,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321196,"statfs":{"total":96636764160,"available":96584904704,"internally_reserved":0,"allocated":19931136,"data_stored":16572899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297060,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53660,"kb_used_data":16008,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94318180,"statfs":{"total":96636764160,"available":96581816320,"internally_reserved":0,"allocated":16392192,"data_stored":13033955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50192,"kb_used_data":18500,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321648,"statfs":{"total":96636764160,"available":96585367552,"internally_reserved":0,"allocated":18944000,"data_stored":15565360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:14:02.992 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:14:03.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:03.107+0000 7f3e8a4cd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:03.119+0000 7f3e8a4cd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:03.296 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:14:03.296 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:14:03.314 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1054,"stamp":"2024-03-30T13:14:02.647157+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":334975796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":417336,"kb_used_data":142328,"kb_used_omap":51,"kb_used_meta":274828,"kb_avail":754557384,"statfs":{"total":773094113280,"available":772666761216,"internally_reserved":0,"allocated":145743872,"data_stored":118803165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52659,"internal_metadata":281424461},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.051882"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1145,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:00.658623+0000","last_change":"2024-03-30T13:13:47.573184+0000","last_active":"2024-03-30T13:14:00.658623+0000","last_peered":"2024-03-30T13:14:00.658623+0000","last_clean":"2024-03-30T13:14:00.658623+0000","last_became_active":"2024-03-30T13:13:47.489523+0000","last_became_peered":"2024-03-30T13:13:47.489523+0000","last_unstale":"2024-03-30T13:14:00.658623+0000","last_undegraded":"2024-03-30T13:14:00.658623+0000","last_fullsized":"2024-03-30T13:14:00.658623+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:12:53.896535+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:12:53.896535+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:16.650956+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18352980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1078,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-03-30T13:13:58.597583+0000","last_change":"2024-03-30T13:13:42.391739+0000","last_active":"2024-03-30T13:13:58.597583+0000","last_peered":"2024-03-30T13:13:58.597583+0000","last_clean":"2024-03-30T13:13:58.597583+0000","last_became_active":"2024-03-30T13:12:00.216256+0000","last_became_peered":"2024-03-30T13:12:00.216256+0000","last_unstale":"2024-03-30T13:13:58.597583+0000","last_undegraded":"2024-03-30T13:13:58.597583+0000","last_fullsized":"2024-03-30T13:13:58.597583+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:42.391496+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_clean_scrub_stamp":"2024-03-30T13:13:42.391496+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:48.275941+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":689,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-03-30T13:13:57.582197+0000","last_change":"2024-03-30T13:13:28.025038+0000","last_active":"2024-03-30T13:13:57.582197+0000","last_peered":"2024-03-30T13:13:57.582197+0000","last_clean":"2024-03-30T13:13:57.582197+0000","last_became_active":"2024-03-30T13:12:51.212130+0000","last_became_peered":"2024-03-30T13:12:51.212130+0000","last_unstale":"2024-03-30T13:13:57.582197+0000","last_undegraded":"2024-03-30T13:13:57.582197+0000","last_fullsized":"2024-03-30T13:13:57.582197+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:13:28.024596+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:13:28.024596+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:47.470109+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":631,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-03-30T13:13:57.582088+0000","last_change":"2024-03-30T13:13:47.119549+0000","last_active":"2024-03-30T13:13:57.582088+0000","last_peered":"2024-03-30T13:13:57.582088+0000","last_clean":"2024-03-30T13:13:57.582088+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:13:57.582088+0000","last_undegraded":"2024-03-30T13:13:57.582088+0000","last_fullsized":"2024-03-30T13:13:57.582088+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:13:47.119125+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:13:47.119125+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:15.963858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":49,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:01.048663+0000","last_change":"2024-03-30T13:13:49.332620+0000","last_active":"2024-03-30T13:14:01.048663+0000","last_peered":"2024-03-30T13:14:01.048663+0000","last_clean":"2024-03-30T13:14:01.048663+0000","last_became_active":"2024-03-30T13:13:47.479287+0000","last_became_peered":"2024-03-30T13:13:47.479287+0000","last_unstale":"2024-03-30T13:14:01.048663+0000","last_undegraded":"2024-03-30T13:14:01.048663+0000","last_fullsized":"2024-03-30T13:14:01.048663+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:49.332375+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:13:49.332375+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:05.013702+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1151,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:01.046998+0000","last_change":"2024-03-30T13:13:48.841791+0000","last_active":"2024-03-30T13:14:01.046998+0000","last_peered":"2024-03-30T13:14:01.046998+0000","last_clean":"2024-03-30T13:14:01.046998+0000","last_became_active":"2024-03-30T13:13:47.486556+0000","last_became_peered":"2024-03-30T13:13:47.486556+0000","last_unstale":"2024-03-30T13:14:01.046998+0000","last_undegraded":"2024-03-30T13:14:01.046998+0000","last_fullsized":"2024-03-30T13:14:01.046998+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:13:35.802156+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:13:35.802156+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:49.445337+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22789962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":666,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-03-30T13:13:57.582146+0000","last_change":"2024-03-30T13:13:16.070252+0000","last_active":"2024-03-30T13:13:57.582146+0000","last_peered":"2024-03-30T13:13:57.582146+0000","last_clean":"2024-03-30T13:13:57.582146+0000","last_became_active":"2024-03-30T13:12:00.202949+0000","last_became_peered":"2024-03-30T13:12:00.202949+0000","last_unstale":"2024-03-30T13:13:57.582146+0000","last_undegraded":"2024-03-30T13:13:57.582146+0000","last_fullsized":"2024-03-30T13:13:57.582146+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:13:16.069976+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:13:16.069976+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:25.894866+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":412,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-03-30T13:13:59.597660+0000","last_change":"2024-03-30T13:13:48.760543+0000","last_active":"2024-03-30T13:13:59.597660+0000","last_peered":"2024-03-30T13:13:59.597660+0000","last_clean":"2024-03-30T13:13:59.597660+0000","last_became_active":"2024-03-30T13:13:47.499413+0000","last_became_peered":"2024-03-30T13:13:47.499413+0000","last_unstale":"2024-03-30T13:13:59.597660+0000","last_undegraded":"2024-03-30T13:13:59.597660+0000","last_fullsized":"2024-03-30T13:13:59.597660+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:13:15.593589+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:13:15.593589+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:35.673056+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1257,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:01.046711+0000","last_change":"2024-03-30T13:13:47.536108+0000","last_active":"2024-03-30T13:14:01.046711+0000","last_peered":"2024-03-30T13:14:01.046711+0000","last_clean":"2024-03-30T13:14:01.046711+0000","last_became_active":"2024-03-30T13:13:47.486649+0000","last_became_peered":"2024-03-30T13:13:47.486649+0000","last_unstale":"2024-03-30T13:14:01.046711+0000","last_undegraded":"2024-03-30T13:14:01.046711+0000","last_fullsized":"2024-03-30T13:14:01.046711+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:13:12.879768+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:13:12.879768+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:31.285371+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":65983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":952,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-03-30T13:13:58.597508+0000","last_change":"2024-03-30T13:13:38.374659+0000","last_active":"2024-03-30T13:13:58.597508+0000","last_peered":"2024-03-30T13:13:58.597508+0000","last_clean":"2024-03-30T13:13:58.597508+0000","last_became_active":"2024-03-30T13:12:00.203605+0000","last_became_peered":"2024-03-30T13:12:00.203605+0000","last_unstale":"2024-03-30T13:13:58.597508+0000","last_undegraded":"2024-03-30T13:13:58.597508+0000","last_fullsized":"2024-03-30T13:13:58.597508+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:38.374502+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:13:38.374502+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:47.293366+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1143,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:00.658439+0000","last_change":"2024-03-30T13:13:57.067928+0000","last_active":"2024-03-30T13:14:00.658439+0000","last_peered":"2024-03-30T13:14:00.658439+0000","last_clean":"2024-03-30T13:14:00.658439+0000","last_became_active":"2024-03-30T13:12:00.202952+0000","last_became_peered":"2024-03-30T13:12:00.202952+0000","last_unstale":"2024-03-30T13:14:00.658439+0000","last_undegraded":"2024-03-30T13:14:00.658439+0000","last_fullsized":"2024-03-30T13:14:00.658439+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:13:57.067688+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:13:57.067688+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:13.208052+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1231,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:00.658433+0000","last_change":"2024-03-30T13:13:48.914104+0000","last_active":"2024-03-30T13:14:00.658433+0000","last_peered":"2024-03-30T13:14:00.658433+0000","last_clean":"2024-03-30T13:14:00.658433+0000","last_became_active":"2024-03-30T13:13:47.498724+0000","last_became_peered":"2024-03-30T13:13:47.498724+0000","last_unstale":"2024-03-30T13:14:00.658433+0000","last_undegraded":"2024-03-30T13:14:00.658433+0000","last_fullsized":"2024-03-30T13:14:00.658433+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:13:48.913862+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:13:48.913862+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:01.275068+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":57731253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":55,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:01.048630+0000","last_change":"2024-03-30T13:13:52.332055+0000","last_active":"2024-03-30T13:14:01.048630+0000","last_peered":"2024-03-30T13:14:01.048630+0000","last_clean":"2024-03-30T13:14:01.048630+0000","last_became_active":"2024-03-30T13:13:47.483197+0000","last_became_peered":"2024-03-30T13:13:47.483197+0000","last_unstale":"2024-03-30T13:14:01.048630+0000","last_undegraded":"2024-03-30T13:14:01.048630+0000","last_fullsized":"2024-03-30T13:14:01.048630+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:13:52.331764+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:13:52.331764+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:15.424847+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":602,"reported_epoch":501,"state":"active+clean","last_fresh":"2024-03-30T13:13:57.582113+0000","last_change":"2024-03-30T13:13:29.018285+0000","last_active":"2024-03-30T13:13:57.582113+0000","last_peered":"2024-03-30T13:13:57.582113+0000","last_clean":"2024-03-30T13:13:57.582113+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:13:57.582113+0000","last_undegraded":"2024-03-30T13:13:57.582113+0000","last_fullsized":"2024-03-30T13:13:57.582113+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:29.018031+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:13:29.018031+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:47.121728+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":47,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:01.048681+0000","last_change":"2024-03-30T13:13:47.578888+0000","last_active":"2024-03-30T13:14:01.048681+0000","last_peered":"2024-03-30T13:14:01.048681+0000","last_clean":"2024-03-30T13:14:01.048681+0000","last_became_active":"2024-03-30T13:13:47.496794+0000","last_became_peered":"2024-03-30T13:13:47.496794+0000","last_unstale":"2024-03-30T13:14:01.048681+0000","last_undegraded":"2024-03-30T13:14:01.048681+0000","last_fullsized":"2024-03-30T13:14:01.048681+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:13:05.217766+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:13:05.217766+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:07.019567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":357,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-03-30T13:13:59.597762+0000","last_change":"2024-03-30T13:13:47.536417+0000","last_active":"2024-03-30T13:13:59.597762+0000","last_peered":"2024-03-30T13:13:59.597762+0000","last_clean":"2024-03-30T13:13:59.597762+0000","last_became_active":"2024-03-30T13:13:47.481478+0000","last_became_peered":"2024-03-30T13:13:47.481478+0000","last_unstale":"2024-03-30T13:13:59.597762+0000","last_undegraded":"2024-03-30T13:13:59.597762+0000","last_fullsized":"2024-03-30T13:13:59.597762+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:12:45.634758+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:12:45.634758+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:14.539491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10453822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":334975796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834169,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53256,"kb_used_data":21108,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94318584,"statfs":{"total":96636764160,"available":96582230016,"internally_reserved":0,"allocated":21614592,"data_stored":18235952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":488,"seq":2095944040454,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63048,"kb_used_data":25852,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94308792,"statfs":{"total":96636764160,"available":96572203008,"internally_reserved":0,"allocated":26472448,"data_stored":23093808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9352,"internal_metadata":38067064},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":441,"seq":1894080577562,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54724,"kb_used_data":17520,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94317116,"statfs":{"total":96636764160,"available":96580726784,"internally_reserved":0,"allocated":17940480,"data_stored":14582243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9343,"internal_metadata":38067073},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509665,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47648,"kb_used_data":12300,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94324192,"statfs":{"total":96636764160,"available":96587972608,"internally_reserved":0,"allocated":12595200,"data_stored":9230162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":461,"seq":1979979923471,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44164,"kb_used_data":11576,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94327676,"statfs":{"total":96636764160,"available":96591540224,"internally_reserved":0,"allocated":11853824,"data_stored":8488786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7692,"internal_metadata":33350132},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509666,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50644,"kb_used_data":19464,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321196,"statfs":{"total":96636764160,"available":96584904704,"internally_reserved":0,"allocated":19931136,"data_stored":16572899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297060,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53660,"kb_used_data":16008,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94318180,"statfs":{"total":96636764160,"available":96581816320,"internally_reserved":0,"allocated":16392192,"data_stored":13033955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50192,"kb_used_data":18500,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321648,"statfs":{"total":96636764160,"available":96585367552,"internally_reserved":0,"allocated":18944000,"data_stored":15565360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:14:03.316 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:14:03.316 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:14:03.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:03.423+0000 7fc65598d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:03.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:03.439+0000 7fc65598d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:03.621 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:14:03.621 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":506,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:14:02.641242+0000","last_up_change":"2024-03-30T13:13:41.420550+0000","last_in_change":"2024-03-30T13:13:46.451253+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":436,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":346,"last_clean_end":457,"up_from":461,"up_thru":490,"down_at":458,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2034332759},{"type":"v1","addr":"172.21.15.67:6818","nonce":2034332759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2034332759},{"type":"v1","addr":"172.21.15.67:6821","nonce":2034332759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2034332759},{"type":"v1","addr":"172.21.15.67:6828","nonce":2034332759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2034332759},{"type":"v1","addr":"172.21.15.67:6825","nonce":2034332759}]},"public_addr":"172.21.15.67:6818/2034332759","cluster_addr":"172.21.15.67:6821/2034332759","heartbeat_back_addr":"172.21.15.67:6828/2034332759","heartbeat_front_addr":"172.21.15.67:6825/2034332759","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":479,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":388,"up_from":393,"up_thru":490,"down_at":389,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6801","nonce":3837194718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6803","nonce":3837194718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6807","nonce":3837194718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6805","nonce":3837194718}]},"public_addr":"172.21.15.67:6801/3837194718","cluster_addr":"172.21.15.67:6803/3837194718","heartbeat_back_addr":"172.21.15.67:6807/3837194718","heartbeat_front_addr":"172.21.15.67:6805/3837194718","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":250,"last_clean_end":435,"up_from":441,"up_thru":480,"down_at":436,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3152653542},{"type":"v1","addr":"172.21.15.82:6803","nonce":3152653542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":3152653542},{"type":"v1","addr":"172.21.15.82:6807","nonce":3152653542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":3152653542},{"type":"v1","addr":"172.21.15.82:6815","nonce":3152653542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":3152653542},{"type":"v1","addr":"172.21.15.82:6811","nonce":3152653542}]},"public_addr":"172.21.15.82:6803/3152653542","cluster_addr":"172.21.15.82:6807/3152653542","heartbeat_back_addr":"172.21.15.82:6815/3152653542","heartbeat_front_addr":"172.21.15.82:6811/3152653542","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":320,"last_clean_end":478,"up_from":488,"up_thru":490,"down_at":479,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":1839650812},{"type":"v1","addr":"172.21.15.82:6817","nonce":1839650812}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":1839650812},{"type":"v1","addr":"172.21.15.82:6819","nonce":1839650812}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":1839650812},{"type":"v1","addr":"172.21.15.82:6823","nonce":1839650812}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":1839650812},{"type":"v1","addr":"172.21.15.82:6821","nonce":1839650812}]},"public_addr":"172.21.15.82:6817/1839650812","cluster_addr":"172.21.15.82:6819/1839650812","heartbeat_back_addr":"172.21.15.82:6823/1839650812","heartbeat_front_addr":"172.21.15.82:6821/1839650812","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":227,"last_clean_end":363,"up_from":372,"up_thru":490,"down_at":364,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6801","nonce":189641108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6806","nonce":189641108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6814","nonce":189641108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6810","nonce":189641108}]},"public_addr":"172.21.15.82:6801/189641108","cluster_addr":"172.21.15.82:6806/189641108","heartbeat_back_addr":"172.21.15.82:6814/189641108","heartbeat_front_addr":"172.21.15.82:6810/189641108","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":298,"last_clean_end":409,"up_from":419,"up_thru":461,"down_at":410,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6825","nonce":2166599911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6827","nonce":2166599911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6831","nonce":2166599911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6829","nonce":2166599911}]},"public_addr":"172.21.15.82:6825/2166599911","cluster_addr":"172.21.15.82:6827/2166599911","heartbeat_back_addr":"172.21.15.82:6831/2166599911","heartbeat_front_addr":"172.21.15.82:6829/2166599911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:13:15.991900+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:13:16.476015+0000","dead_epoch":458},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:13:10.942048+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:13:12.350627+0000","dead_epoch":389},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:13:24.846119+0000","dead_epoch":436},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:13:40.488725+0000","dead_epoch":479},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:13:00.706194+0000","dead_epoch":364},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:13:07.171435+0000","dead_epoch":410}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:14:03.640 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:14:03.641 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:14:03.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:03.763+0000 7f23e64a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:03.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:03.783+0000 7f23e64a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:03.958 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:14:03.978 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:14:04.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:04.083+0000 7f1a7f9fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:04.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:04.103+0000 7f1a7f9fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:04.283 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:14:04.283 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:14:04.302 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:14:04.302 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:14:04.302 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:14:04.302 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:14:04.302 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 6, 3, 7, 4, 1, 5] 2024-03-30T13:14:04.302 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:14:04.303 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:14:04.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:04.411+0000 7f571f04d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:04.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:04.423+0000 7f571f04d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:04.598 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:14:04.599 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:14:04.615 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1055,"stamp":"2024-03-30T13:14:04.102947+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":334975796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":417396,"kb_used_data":142388,"kb_used_omap":51,"kb_used_meta":274828,"kb_avail":754557324,"statfs":{"total":773094113280,"available":772666699776,"internally_reserved":0,"allocated":145805312,"data_stored":118830600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52659,"internal_metadata":281424461},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.504921"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1145,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:00.658623+0000","last_change":"2024-03-30T13:13:47.573184+0000","last_active":"2024-03-30T13:14:00.658623+0000","last_peered":"2024-03-30T13:14:00.658623+0000","last_clean":"2024-03-30T13:14:00.658623+0000","last_became_active":"2024-03-30T13:13:47.489523+0000","last_became_peered":"2024-03-30T13:13:47.489523+0000","last_unstale":"2024-03-30T13:14:00.658623+0000","last_undegraded":"2024-03-30T13:14:00.658623+0000","last_fullsized":"2024-03-30T13:14:00.658623+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:12:53.896535+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:12:53.896535+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:16.650956+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18352980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1078,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-03-30T13:13:58.597583+0000","last_change":"2024-03-30T13:13:42.391739+0000","last_active":"2024-03-30T13:13:58.597583+0000","last_peered":"2024-03-30T13:13:58.597583+0000","last_clean":"2024-03-30T13:13:58.597583+0000","last_became_active":"2024-03-30T13:12:00.216256+0000","last_became_peered":"2024-03-30T13:12:00.216256+0000","last_unstale":"2024-03-30T13:13:58.597583+0000","last_undegraded":"2024-03-30T13:13:58.597583+0000","last_fullsized":"2024-03-30T13:13:58.597583+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:42.391496+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_clean_scrub_stamp":"2024-03-30T13:13:42.391496+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:48.275941+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":699,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-03-30T13:14:03.112162+0000","last_change":"2024-03-30T13:13:28.025038+0000","last_active":"2024-03-30T13:14:03.112162+0000","last_peered":"2024-03-30T13:14:03.112162+0000","last_clean":"2024-03-30T13:14:03.112162+0000","last_became_active":"2024-03-30T13:12:51.212130+0000","last_became_peered":"2024-03-30T13:12:51.212130+0000","last_unstale":"2024-03-30T13:14:03.112162+0000","last_undegraded":"2024-03-30T13:14:03.112162+0000","last_fullsized":"2024-03-30T13:14:03.112162+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:13:28.024596+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:13:28.024596+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:47.470109+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":641,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-03-30T13:14:03.112068+0000","last_change":"2024-03-30T13:13:47.119549+0000","last_active":"2024-03-30T13:14:03.112068+0000","last_peered":"2024-03-30T13:14:03.112068+0000","last_clean":"2024-03-30T13:14:03.112068+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:14:03.112068+0000","last_undegraded":"2024-03-30T13:14:03.112068+0000","last_fullsized":"2024-03-30T13:14:03.112068+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:13:47.119125+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:13:47.119125+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:15.963858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":49,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:01.048663+0000","last_change":"2024-03-30T13:13:49.332620+0000","last_active":"2024-03-30T13:14:01.048663+0000","last_peered":"2024-03-30T13:14:01.048663+0000","last_clean":"2024-03-30T13:14:01.048663+0000","last_became_active":"2024-03-30T13:13:47.479287+0000","last_became_peered":"2024-03-30T13:13:47.479287+0000","last_unstale":"2024-03-30T13:14:01.048663+0000","last_undegraded":"2024-03-30T13:14:01.048663+0000","last_fullsized":"2024-03-30T13:14:01.048663+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:49.332375+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:13:49.332375+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:05.013702+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1151,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:01.046998+0000","last_change":"2024-03-30T13:13:48.841791+0000","last_active":"2024-03-30T13:14:01.046998+0000","last_peered":"2024-03-30T13:14:01.046998+0000","last_clean":"2024-03-30T13:14:01.046998+0000","last_became_active":"2024-03-30T13:13:47.486556+0000","last_became_peered":"2024-03-30T13:13:47.486556+0000","last_unstale":"2024-03-30T13:14:01.046998+0000","last_undegraded":"2024-03-30T13:14:01.046998+0000","last_fullsized":"2024-03-30T13:14:01.046998+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:13:35.802156+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:13:35.802156+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:49.445337+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22789962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":676,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-03-30T13:14:03.112096+0000","last_change":"2024-03-30T13:13:16.070252+0000","last_active":"2024-03-30T13:14:03.112096+0000","last_peered":"2024-03-30T13:14:03.112096+0000","last_clean":"2024-03-30T13:14:03.112096+0000","last_became_active":"2024-03-30T13:12:00.202949+0000","last_became_peered":"2024-03-30T13:12:00.202949+0000","last_unstale":"2024-03-30T13:14:03.112096+0000","last_undegraded":"2024-03-30T13:14:03.112096+0000","last_fullsized":"2024-03-30T13:14:03.112096+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:13:16.069976+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:13:16.069976+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:25.894866+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":412,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-03-30T13:13:59.597660+0000","last_change":"2024-03-30T13:13:48.760543+0000","last_active":"2024-03-30T13:13:59.597660+0000","last_peered":"2024-03-30T13:13:59.597660+0000","last_clean":"2024-03-30T13:13:59.597660+0000","last_became_active":"2024-03-30T13:13:47.499413+0000","last_became_peered":"2024-03-30T13:13:47.499413+0000","last_unstale":"2024-03-30T13:13:59.597660+0000","last_undegraded":"2024-03-30T13:13:59.597660+0000","last_fullsized":"2024-03-30T13:13:59.597660+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:13:15.593589+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:13:15.593589+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:35.673056+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1257,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:01.046711+0000","last_change":"2024-03-30T13:13:47.536108+0000","last_active":"2024-03-30T13:14:01.046711+0000","last_peered":"2024-03-30T13:14:01.046711+0000","last_clean":"2024-03-30T13:14:01.046711+0000","last_became_active":"2024-03-30T13:13:47.486649+0000","last_became_peered":"2024-03-30T13:13:47.486649+0000","last_unstale":"2024-03-30T13:14:01.046711+0000","last_undegraded":"2024-03-30T13:14:01.046711+0000","last_fullsized":"2024-03-30T13:14:01.046711+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:13:12.879768+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:13:12.879768+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:31.285371+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":65983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":952,"reported_epoch":502,"state":"active+clean","last_fresh":"2024-03-30T13:13:58.597508+0000","last_change":"2024-03-30T13:13:38.374659+0000","last_active":"2024-03-30T13:13:58.597508+0000","last_peered":"2024-03-30T13:13:58.597508+0000","last_clean":"2024-03-30T13:13:58.597508+0000","last_became_active":"2024-03-30T13:12:00.203605+0000","last_became_peered":"2024-03-30T13:12:00.203605+0000","last_unstale":"2024-03-30T13:13:58.597508+0000","last_undegraded":"2024-03-30T13:13:58.597508+0000","last_fullsized":"2024-03-30T13:13:58.597508+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:38.374502+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:13:38.374502+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:47.293366+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1143,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:00.658439+0000","last_change":"2024-03-30T13:13:57.067928+0000","last_active":"2024-03-30T13:14:00.658439+0000","last_peered":"2024-03-30T13:14:00.658439+0000","last_clean":"2024-03-30T13:14:00.658439+0000","last_became_active":"2024-03-30T13:12:00.202952+0000","last_became_peered":"2024-03-30T13:12:00.202952+0000","last_unstale":"2024-03-30T13:14:00.658439+0000","last_undegraded":"2024-03-30T13:14:00.658439+0000","last_fullsized":"2024-03-30T13:14:00.658439+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:13:57.067688+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:13:57.067688+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:13.208052+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1231,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:00.658433+0000","last_change":"2024-03-30T13:13:48.914104+0000","last_active":"2024-03-30T13:14:00.658433+0000","last_peered":"2024-03-30T13:14:00.658433+0000","last_clean":"2024-03-30T13:14:00.658433+0000","last_became_active":"2024-03-30T13:13:47.498724+0000","last_became_peered":"2024-03-30T13:13:47.498724+0000","last_unstale":"2024-03-30T13:14:00.658433+0000","last_undegraded":"2024-03-30T13:14:00.658433+0000","last_fullsized":"2024-03-30T13:14:00.658433+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:13:48.913862+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:13:48.913862+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:01.275068+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":57731253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":55,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:01.048630+0000","last_change":"2024-03-30T13:13:52.332055+0000","last_active":"2024-03-30T13:14:01.048630+0000","last_peered":"2024-03-30T13:14:01.048630+0000","last_clean":"2024-03-30T13:14:01.048630+0000","last_became_active":"2024-03-30T13:13:47.483197+0000","last_became_peered":"2024-03-30T13:13:47.483197+0000","last_unstale":"2024-03-30T13:14:01.048630+0000","last_undegraded":"2024-03-30T13:14:01.048630+0000","last_fullsized":"2024-03-30T13:14:01.048630+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:13:52.331764+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:13:52.331764+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:15.424847+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":612,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-03-30T13:14:03.112409+0000","last_change":"2024-03-30T13:13:29.018285+0000","last_active":"2024-03-30T13:14:03.112409+0000","last_peered":"2024-03-30T13:14:03.112409+0000","last_clean":"2024-03-30T13:14:03.112409+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:14:03.112409+0000","last_undegraded":"2024-03-30T13:14:03.112409+0000","last_fullsized":"2024-03-30T13:14:03.112409+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:29.018031+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:13:29.018031+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:47.121728+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":47,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:01.048681+0000","last_change":"2024-03-30T13:13:47.578888+0000","last_active":"2024-03-30T13:14:01.048681+0000","last_peered":"2024-03-30T13:14:01.048681+0000","last_clean":"2024-03-30T13:14:01.048681+0000","last_became_active":"2024-03-30T13:13:47.496794+0000","last_became_peered":"2024-03-30T13:13:47.496794+0000","last_unstale":"2024-03-30T13:14:01.048681+0000","last_undegraded":"2024-03-30T13:14:01.048681+0000","last_fullsized":"2024-03-30T13:14:01.048681+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:13:05.217766+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:13:05.217766+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:07.019567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":357,"reported_epoch":503,"state":"active+clean","last_fresh":"2024-03-30T13:13:59.597762+0000","last_change":"2024-03-30T13:13:47.536417+0000","last_active":"2024-03-30T13:13:59.597762+0000","last_peered":"2024-03-30T13:13:59.597762+0000","last_clean":"2024-03-30T13:13:59.597762+0000","last_became_active":"2024-03-30T13:13:47.481478+0000","last_became_peered":"2024-03-30T13:13:47.481478+0000","last_unstale":"2024-03-30T13:13:59.597762+0000","last_undegraded":"2024-03-30T13:13:59.597762+0000","last_fullsized":"2024-03-30T13:13:59.597762+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:12:45.634758+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:12:45.634758+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:14.539491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10453822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":334975796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834169,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53256,"kb_used_data":21108,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94318584,"statfs":{"total":96636764160,"available":96582230016,"internally_reserved":0,"allocated":21614592,"data_stored":18235952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":488,"seq":2095944040454,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63048,"kb_used_data":25852,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94308792,"statfs":{"total":96636764160,"available":96572203008,"internally_reserved":0,"allocated":26472448,"data_stored":23093808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9352,"internal_metadata":38067064},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":441,"seq":1894080577562,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54724,"kb_used_data":17520,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94317116,"statfs":{"total":96636764160,"available":96580726784,"internally_reserved":0,"allocated":17940480,"data_stored":14582243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9343,"internal_metadata":38067073},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509665,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47648,"kb_used_data":12300,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94324192,"statfs":{"total":96636764160,"available":96587972608,"internally_reserved":0,"allocated":12595200,"data_stored":9230162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":461,"seq":1979979923471,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44164,"kb_used_data":11576,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94327676,"statfs":{"total":96636764160,"available":96591540224,"internally_reserved":0,"allocated":11853824,"data_stored":8488786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7692,"internal_metadata":33350132},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509666,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50644,"kb_used_data":19464,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321196,"statfs":{"total":96636764160,"available":96584904704,"internally_reserved":0,"allocated":19931136,"data_stored":16572899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297061,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53720,"kb_used_data":16068,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94318120,"statfs":{"total":96636764160,"available":96581754880,"internally_reserved":0,"allocated":16453632,"data_stored":13061390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50192,"kb_used_data":18500,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321648,"statfs":{"total":96636764160,"available":96585367552,"internally_reserved":0,"allocated":18944000,"data_stored":15565360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:14:04.616 INFO:tasks.thrashosds.thrasher:Choosing PG 1.1 with acting set [5, 2, 0] 2024-03-30T13:14:04.616 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 2, 6, 3, 7, 4, 1, 5] 2024-03-30T13:14:04.616 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-30T13:14:04.616 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:14:04.671 INFO:tasks.ceph.osd.5:Stopped 2024-03-30T13:14:04.671 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 2, 6, 3, 7, 4, 1, 5] 2024-03-30T13:14:04.671 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 5 2024-03-30T13:14:04.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:04.783+0000 7f8bbd59b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:04.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:04.799+0000 7f8bbd59b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:04.986 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:14:04.983+0000 7fbb38f08640 -1 mon.a@0(leader).osd e506 definitely_dead 0 2024-03-30T13:14:05.668 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.5. 2024-03-30T13:14:05.698 INFO:tasks.thrashosds.thrasher:dead_osds=[5], live_osds=[0, 2, 6, 3, 7, 4, 1] 2024-03-30T13:14:05.698 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:14:05.698 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:14:05.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:05.811+0000 7f47d7f20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:05.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:05.827+0000 7f47d7f20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:06.001 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:14:06.001 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:14:06.019 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1056,"stamp":"2024-03-30T13:14:05.660884+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":334975796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":354564,"kb_used_data":116752,"kb_used_omap":42,"kb_used_meta":237653,"kb_avail":660248316,"statfs":{"total":676457349120,"available":676094275584,"internally_reserved":0,"allocated":119554048,"data_stored":95835558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43307,"internal_metadata":243357397},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.559042"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1145,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:00.658623+0000","last_change":"2024-03-30T13:13:47.573184+0000","last_active":"2024-03-30T13:14:00.658623+0000","last_peered":"2024-03-30T13:14:00.658623+0000","last_clean":"2024-03-30T13:14:00.658623+0000","last_became_active":"2024-03-30T13:13:47.489523+0000","last_became_peered":"2024-03-30T13:13:47.489523+0000","last_unstale":"2024-03-30T13:14:00.658623+0000","last_undegraded":"2024-03-30T13:14:00.658623+0000","last_fullsized":"2024-03-30T13:14:00.658623+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:12:53.896535+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:12:53.896535+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:16.650956+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18352980,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1086,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-03-30T13:14:03.112514+0000","last_change":"2024-03-30T13:13:42.391739+0000","last_active":"2024-03-30T13:14:03.112514+0000","last_peered":"2024-03-30T13:14:03.112514+0000","last_clean":"2024-03-30T13:14:03.112514+0000","last_became_active":"2024-03-30T13:12:00.216256+0000","last_became_peered":"2024-03-30T13:12:00.216256+0000","last_unstale":"2024-03-30T13:14:03.112514+0000","last_undegraded":"2024-03-30T13:14:03.112514+0000","last_fullsized":"2024-03-30T13:14:03.112514+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:42.391496+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_clean_scrub_stamp":"2024-03-30T13:13:42.391496+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:48.275941+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":699,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-03-30T13:14:03.112162+0000","last_change":"2024-03-30T13:13:28.025038+0000","last_active":"2024-03-30T13:14:03.112162+0000","last_peered":"2024-03-30T13:14:03.112162+0000","last_clean":"2024-03-30T13:14:03.112162+0000","last_became_active":"2024-03-30T13:12:51.212130+0000","last_became_peered":"2024-03-30T13:12:51.212130+0000","last_unstale":"2024-03-30T13:14:03.112162+0000","last_undegraded":"2024-03-30T13:14:03.112162+0000","last_fullsized":"2024-03-30T13:14:03.112162+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:13:28.024596+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:13:28.024596+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:47.470109+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":641,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-03-30T13:14:03.112068+0000","last_change":"2024-03-30T13:13:47.119549+0000","last_active":"2024-03-30T13:14:03.112068+0000","last_peered":"2024-03-30T13:14:03.112068+0000","last_clean":"2024-03-30T13:14:03.112068+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:14:03.112068+0000","last_undegraded":"2024-03-30T13:14:03.112068+0000","last_fullsized":"2024-03-30T13:14:03.112068+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:13:47.119125+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:13:47.119125+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:15.963858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":49,"reported_epoch":504,"state":"stale+active+clean","last_fresh":"2024-03-30T13:14:01.048663+0000","last_change":"2024-03-30T13:13:49.332620+0000","last_active":"2024-03-30T13:14:01.048663+0000","last_peered":"2024-03-30T13:14:01.048663+0000","last_clean":"2024-03-30T13:14:01.048663+0000","last_became_active":"2024-03-30T13:13:47.479287+0000","last_became_peered":"2024-03-30T13:13:47.479287+0000","last_unstale":"2024-03-30T13:14:05.660906+0000","last_undegraded":"2024-03-30T13:14:01.048663+0000","last_fullsized":"2024-03-30T13:14:01.048663+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:49.332375+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:13:49.332375+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:05.013702+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1151,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:01.046998+0000","last_change":"2024-03-30T13:13:48.841791+0000","last_active":"2024-03-30T13:14:01.046998+0000","last_peered":"2024-03-30T13:14:01.046998+0000","last_clean":"2024-03-30T13:14:01.046998+0000","last_became_active":"2024-03-30T13:13:47.486556+0000","last_became_peered":"2024-03-30T13:13:47.486556+0000","last_unstale":"2024-03-30T13:14:01.046998+0000","last_undegraded":"2024-03-30T13:14:01.046998+0000","last_fullsized":"2024-03-30T13:14:01.046998+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:13:35.802156+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:13:35.802156+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:49.445337+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22789962,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":676,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-03-30T13:14:03.112096+0000","last_change":"2024-03-30T13:13:16.070252+0000","last_active":"2024-03-30T13:14:03.112096+0000","last_peered":"2024-03-30T13:14:03.112096+0000","last_clean":"2024-03-30T13:14:03.112096+0000","last_became_active":"2024-03-30T13:12:00.202949+0000","last_became_peered":"2024-03-30T13:12:00.202949+0000","last_unstale":"2024-03-30T13:14:03.112096+0000","last_undegraded":"2024-03-30T13:14:03.112096+0000","last_fullsized":"2024-03-30T13:14:03.112096+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:13:16.069976+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:13:16.069976+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:25.894866+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":418,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-03-30T13:14:02.646977+0000","last_change":"2024-03-30T13:13:48.760543+0000","last_active":"2024-03-30T13:14:02.646977+0000","last_peered":"2024-03-30T13:14:02.646977+0000","last_clean":"2024-03-30T13:14:02.646977+0000","last_became_active":"2024-03-30T13:13:47.499413+0000","last_became_peered":"2024-03-30T13:13:47.499413+0000","last_unstale":"2024-03-30T13:14:02.646977+0000","last_undegraded":"2024-03-30T13:14:02.646977+0000","last_fullsized":"2024-03-30T13:14:02.646977+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":492,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:13:15.593589+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:13:15.593589+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:35.673056+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1257,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:01.046711+0000","last_change":"2024-03-30T13:13:47.536108+0000","last_active":"2024-03-30T13:14:01.046711+0000","last_peered":"2024-03-30T13:14:01.046711+0000","last_clean":"2024-03-30T13:14:01.046711+0000","last_became_active":"2024-03-30T13:13:47.486649+0000","last_became_peered":"2024-03-30T13:13:47.486649+0000","last_unstale":"2024-03-30T13:14:01.046711+0000","last_undegraded":"2024-03-30T13:14:01.046711+0000","last_fullsized":"2024-03-30T13:14:01.046711+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:13:12.879768+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:13:12.879768+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:31.285371+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":65983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":961,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-03-30T13:14:03.112196+0000","last_change":"2024-03-30T13:13:38.374659+0000","last_active":"2024-03-30T13:14:03.112196+0000","last_peered":"2024-03-30T13:14:03.112196+0000","last_clean":"2024-03-30T13:14:03.112196+0000","last_became_active":"2024-03-30T13:12:00.203605+0000","last_became_peered":"2024-03-30T13:12:00.203605+0000","last_unstale":"2024-03-30T13:14:03.112196+0000","last_undegraded":"2024-03-30T13:14:03.112196+0000","last_fullsized":"2024-03-30T13:14:03.112196+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:38.374502+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:13:38.374502+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:47.293366+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1143,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:00.658439+0000","last_change":"2024-03-30T13:13:57.067928+0000","last_active":"2024-03-30T13:14:00.658439+0000","last_peered":"2024-03-30T13:14:00.658439+0000","last_clean":"2024-03-30T13:14:00.658439+0000","last_became_active":"2024-03-30T13:12:00.202952+0000","last_became_peered":"2024-03-30T13:12:00.202952+0000","last_unstale":"2024-03-30T13:14:00.658439+0000","last_undegraded":"2024-03-30T13:14:00.658439+0000","last_fullsized":"2024-03-30T13:14:00.658439+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:13:57.067688+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:13:57.067688+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:13.208052+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1231,"reported_epoch":504,"state":"active+clean","last_fresh":"2024-03-30T13:14:00.658433+0000","last_change":"2024-03-30T13:13:48.914104+0000","last_active":"2024-03-30T13:14:00.658433+0000","last_peered":"2024-03-30T13:14:00.658433+0000","last_clean":"2024-03-30T13:14:00.658433+0000","last_became_active":"2024-03-30T13:13:47.498724+0000","last_became_peered":"2024-03-30T13:13:47.498724+0000","last_unstale":"2024-03-30T13:14:00.658433+0000","last_undegraded":"2024-03-30T13:14:00.658433+0000","last_fullsized":"2024-03-30T13:14:00.658433+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:13:48.913862+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:13:48.913862+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:01.275068+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":57731253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":55,"reported_epoch":504,"state":"stale+active+clean","last_fresh":"2024-03-30T13:14:01.048630+0000","last_change":"2024-03-30T13:13:52.332055+0000","last_active":"2024-03-30T13:14:01.048630+0000","last_peered":"2024-03-30T13:14:01.048630+0000","last_clean":"2024-03-30T13:14:01.048630+0000","last_became_active":"2024-03-30T13:13:47.483197+0000","last_became_peered":"2024-03-30T13:13:47.483197+0000","last_unstale":"2024-03-30T13:14:05.660910+0000","last_undegraded":"2024-03-30T13:14:01.048630+0000","last_fullsized":"2024-03-30T13:14:01.048630+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:13:52.331764+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:13:52.331764+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:15.424847+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":612,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-03-30T13:14:03.112409+0000","last_change":"2024-03-30T13:13:29.018285+0000","last_active":"2024-03-30T13:14:03.112409+0000","last_peered":"2024-03-30T13:14:03.112409+0000","last_clean":"2024-03-30T13:14:03.112409+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:14:03.112409+0000","last_undegraded":"2024-03-30T13:14:03.112409+0000","last_fullsized":"2024-03-30T13:14:03.112409+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:29.018031+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:13:29.018031+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:47.121728+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":47,"reported_epoch":504,"state":"stale+active+clean","last_fresh":"2024-03-30T13:14:01.048681+0000","last_change":"2024-03-30T13:13:47.578888+0000","last_active":"2024-03-30T13:14:01.048681+0000","last_peered":"2024-03-30T13:14:01.048681+0000","last_clean":"2024-03-30T13:14:01.048681+0000","last_became_active":"2024-03-30T13:13:47.496794+0000","last_became_peered":"2024-03-30T13:13:47.496794+0000","last_unstale":"2024-03-30T13:14:05.660912+0000","last_undegraded":"2024-03-30T13:14:01.048681+0000","last_fullsized":"2024-03-30T13:14:01.048681+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:13:05.217766+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:13:05.217766+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:07.019567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":363,"reported_epoch":506,"state":"active+clean","last_fresh":"2024-03-30T13:14:02.646999+0000","last_change":"2024-03-30T13:13:47.536417+0000","last_active":"2024-03-30T13:14:02.646999+0000","last_peered":"2024-03-30T13:14:02.646999+0000","last_clean":"2024-03-30T13:14:02.646999+0000","last_became_active":"2024-03-30T13:13:47.481478+0000","last_became_peered":"2024-03-30T13:13:47.481478+0000","last_unstale":"2024-03-30T13:14:02.646999+0000","last_undegraded":"2024-03-30T13:14:02.646999+0000","last_fullsized":"2024-03-30T13:14:02.646999+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:12:45.634758+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:12:45.634758+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:14.539491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10453822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":334975796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834169,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53256,"kb_used_data":21108,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94318584,"statfs":{"total":96636764160,"available":96582230016,"internally_reserved":0,"allocated":21614592,"data_stored":18235952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":441,"seq":1894080577563,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54780,"kb_used_data":17580,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94317060,"statfs":{"total":96636764160,"available":96580669440,"internally_reserved":0,"allocated":18001920,"data_stored":14609678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9343,"internal_metadata":38067073},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509666,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47692,"kb_used_data":12348,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94324148,"statfs":{"total":96636764160,"available":96587927552,"internally_reserved":0,"allocated":12644352,"data_stored":9252110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":461,"seq":1979979923472,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44216,"kb_used_data":11624,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94327624,"statfs":{"total":96636764160,"available":96591486976,"internally_reserved":0,"allocated":11902976,"data_stored":8510734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7692,"internal_metadata":33350132},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509667,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50708,"kb_used_data":19524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321132,"statfs":{"total":96636764160,"available":96584839168,"internally_reserved":0,"allocated":19992576,"data_stored":16600334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297061,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53720,"kb_used_data":16068,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94318120,"statfs":{"total":96636764160,"available":96581754880,"internally_reserved":0,"allocated":16453632,"data_stored":13061390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50192,"kb_used_data":18500,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321648,"statfs":{"total":96636764160,"available":96585367552,"internally_reserved":0,"allocated":18944000,"data_stored":15565360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:14:06.020 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:14:31.021 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:14:31.021 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:14:31.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:31.147+0000 7fc814460640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:31.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:31.159+0000 7fc814460640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:31.345 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:14:31.345 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:14:31.361 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1075,"stamp":"2024-03-30T13:14:30.184355+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":159,"num_bytes_recovered":387849892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":379060,"kb_used_data":137120,"kb_used_omap":42,"kb_used_meta":241813,"kb_avail":660223820,"statfs":{"total":676457349120,"available":676069191680,"internally_reserved":0,"allocated":140410880,"data_stored":116402894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43307,"internal_metadata":247617237},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.031098"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1179,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-03-30T13:14:22.354485+0000","last_change":"2024-03-30T13:14:07.953630+0000","last_active":"2024-03-30T13:14:22.354485+0000","last_peered":"2024-03-30T13:14:22.354485+0000","last_clean":"2024-03-30T13:14:22.354485+0000","last_became_active":"2024-03-30T13:14:06.681212+0000","last_became_peered":"2024-03-30T13:14:06.681212+0000","last_unstale":"2024-03-30T13:14:22.354485+0000","last_undegraded":"2024-03-30T13:14:22.354485+0000","last_fullsized":"2024-03-30T13:14:22.354485+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:14:07.953391+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:14:07.953391+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:20.561846+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22941225,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1099,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-30T13:14:28.176996+0000","last_change":"2024-03-30T13:13:42.391739+0000","last_active":"2024-03-30T13:14:28.176996+0000","last_peered":"2024-03-30T13:14:28.176996+0000","last_clean":"2024-03-30T13:14:28.176996+0000","last_became_active":"2024-03-30T13:12:00.216256+0000","last_became_peered":"2024-03-30T13:12:00.216256+0000","last_unstale":"2024-03-30T13:14:28.176996+0000","last_undegraded":"2024-03-30T13:14:28.176996+0000","last_fullsized":"2024-03-30T13:14:28.176996+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:42.391496+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_clean_scrub_stamp":"2024-03-30T13:13:42.391496+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:48.275941+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":711,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-30T13:14:28.176609+0000","last_change":"2024-03-30T13:13:28.025038+0000","last_active":"2024-03-30T13:14:28.176609+0000","last_peered":"2024-03-30T13:14:28.176609+0000","last_clean":"2024-03-30T13:14:28.176609+0000","last_became_active":"2024-03-30T13:12:51.212130+0000","last_became_peered":"2024-03-30T13:12:51.212130+0000","last_unstale":"2024-03-30T13:14:28.176609+0000","last_undegraded":"2024-03-30T13:14:28.176609+0000","last_fullsized":"2024-03-30T13:14:28.176609+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:13:28.024596+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:13:28.024596+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:47.470109+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":653,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-30T13:14:28.176505+0000","last_change":"2024-03-30T13:13:47.119549+0000","last_active":"2024-03-30T13:14:28.176505+0000","last_peered":"2024-03-30T13:14:28.176505+0000","last_clean":"2024-03-30T13:14:28.176505+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:14:28.176505+0000","last_undegraded":"2024-03-30T13:14:28.176505+0000","last_fullsized":"2024-03-30T13:14:28.176505+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:13:47.119125+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:13:47.119125+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:15.963858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":23,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-30T13:14:28.176237+0000","last_change":"2024-03-30T13:14:06.764589+0000","last_active":"2024-03-30T13:14:28.176237+0000","last_peered":"2024-03-30T13:14:28.176237+0000","last_clean":"2024-03-30T13:14:28.176237+0000","last_became_active":"2024-03-30T13:14:06.682707+0000","last_became_peered":"2024-03-30T13:14:06.682707+0000","last_unstale":"2024-03-30T13:14:28.176237+0000","last_undegraded":"2024-03-30T13:14:28.176237+0000","last_fullsized":"2024-03-30T13:14:28.176237+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:49.332375+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:13:49.332375+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:54.913571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1183,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-03-30T13:14:22.352042+0000","last_change":"2024-03-30T13:14:06.784379+0000","last_active":"2024-03-30T13:14:22.352042+0000","last_peered":"2024-03-30T13:14:22.352042+0000","last_clean":"2024-03-30T13:14:22.352042+0000","last_became_active":"2024-03-30T13:14:06.682980+0000","last_became_peered":"2024-03-30T13:14:06.682980+0000","last_unstale":"2024-03-30T13:14:22.352042+0000","last_undegraded":"2024-03-30T13:14:22.352042+0000","last_fullsized":"2024-03-30T13:14:22.352042+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:13:35.802156+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:13:35.802156+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:02.286372+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":696,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-30T13:14:28.176568+0000","last_change":"2024-03-30T13:14:26.237658+0000","last_active":"2024-03-30T13:14:28.176568+0000","last_peered":"2024-03-30T13:14:28.176568+0000","last_clean":"2024-03-30T13:14:28.176568+0000","last_became_active":"2024-03-30T13:12:00.202949+0000","last_became_peered":"2024-03-30T13:12:00.202949+0000","last_unstale":"2024-03-30T13:14:28.176568+0000","last_undegraded":"2024-03-30T13:14:28.176568+0000","last_fullsized":"2024-03-30T13:14:28.176568+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:14:26.237251+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:14:26.237251+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:30.208212+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":442,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-03-30T13:14:22.153405+0000","last_change":"2024-03-30T13:14:06.824798+0000","last_active":"2024-03-30T13:14:22.153405+0000","last_peered":"2024-03-30T13:14:22.153405+0000","last_clean":"2024-03-30T13:14:22.153405+0000","last_became_active":"2024-03-30T13:14:06.684981+0000","last_became_peered":"2024-03-30T13:14:06.684981+0000","last_unstale":"2024-03-30T13:14:22.153405+0000","last_undegraded":"2024-03-30T13:14:22.153405+0000","last_fullsized":"2024-03-30T13:14:22.153405+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:13:15.593589+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:13:15.593589+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:38.828592+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1292,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-03-30T13:14:24.557477+0000","last_change":"2024-03-30T13:14:24.557477+0000","last_active":"2024-03-30T13:14:24.557477+0000","last_peered":"2024-03-30T13:14:24.557477+0000","last_clean":"2024-03-30T13:14:24.557477+0000","last_became_active":"2024-03-30T13:14:06.684557+0000","last_became_peered":"2024-03-30T13:14:06.684557+0000","last_unstale":"2024-03-30T13:14:24.557477+0000","last_undegraded":"2024-03-30T13:14:24.557477+0000","last_fullsized":"2024-03-30T13:14:24.557477+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:14:24.557218+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:14:24.557218+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:49.188514+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":71059072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":973,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-30T13:14:28.176957+0000","last_change":"2024-03-30T13:13:38.374659+0000","last_active":"2024-03-30T13:14:28.176957+0000","last_peered":"2024-03-30T13:14:28.176957+0000","last_clean":"2024-03-30T13:14:28.176957+0000","last_became_active":"2024-03-30T13:12:00.203605+0000","last_became_peered":"2024-03-30T13:12:00.203605+0000","last_unstale":"2024-03-30T13:14:28.176957+0000","last_undegraded":"2024-03-30T13:14:28.176957+0000","last_fullsized":"2024-03-30T13:14:28.176957+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:38.374502+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:13:38.374502+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:47.293366+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1157,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-03-30T13:14:22.354329+0000","last_change":"2024-03-30T13:13:57.067928+0000","last_active":"2024-03-30T13:14:22.354329+0000","last_peered":"2024-03-30T13:14:22.354329+0000","last_clean":"2024-03-30T13:14:22.354329+0000","last_became_active":"2024-03-30T13:12:00.202952+0000","last_became_peered":"2024-03-30T13:12:00.202952+0000","last_unstale":"2024-03-30T13:14:22.354329+0000","last_undegraded":"2024-03-30T13:14:22.354329+0000","last_fullsized":"2024-03-30T13:14:22.354329+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:13:57.067688+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:13:57.067688+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:13.208052+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1269,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-03-30T13:14:22.354450+0000","last_change":"2024-03-30T13:14:06.847949+0000","last_active":"2024-03-30T13:14:22.354450+0000","last_peered":"2024-03-30T13:14:22.354450+0000","last_clean":"2024-03-30T13:14:22.354450+0000","last_became_active":"2024-03-30T13:14:06.679208+0000","last_became_peered":"2024-03-30T13:14:06.679208+0000","last_unstale":"2024-03-30T13:14:22.354450+0000","last_undegraded":"2024-03-30T13:14:22.354450+0000","last_fullsized":"2024-03-30T13:14:22.354450+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:13:48.913862+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:13:48.913862+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:13.851311+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":76975004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":27,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-03-30T13:14:22.354170+0000","last_change":"2024-03-30T13:14:06.888086+0000","last_active":"2024-03-30T13:14:22.354170+0000","last_peered":"2024-03-30T13:14:22.354170+0000","last_clean":"2024-03-30T13:14:22.354170+0000","last_became_active":"2024-03-30T13:14:06.681378+0000","last_became_peered":"2024-03-30T13:14:06.681378+0000","last_unstale":"2024-03-30T13:14:22.354170+0000","last_undegraded":"2024-03-30T13:14:22.354170+0000","last_fullsized":"2024-03-30T13:14:22.354170+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:13:52.331764+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:13:52.331764+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:08.794517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":624,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-30T13:14:28.176541+0000","last_change":"2024-03-30T13:13:29.018285+0000","last_active":"2024-03-30T13:14:28.176541+0000","last_peered":"2024-03-30T13:14:28.176541+0000","last_clean":"2024-03-30T13:14:28.176541+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:14:28.176541+0000","last_undegraded":"2024-03-30T13:14:28.176541+0000","last_fullsized":"2024-03-30T13:14:28.176541+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:29.018031+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:13:29.018031+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:47.121728+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":37,"reported_epoch":512,"state":"active+clean","last_fresh":"2024-03-30T13:14:28.176274+0000","last_change":"2024-03-30T13:14:13.517388+0000","last_active":"2024-03-30T13:14:28.176274+0000","last_peered":"2024-03-30T13:14:28.176274+0000","last_clean":"2024-03-30T13:14:28.176274+0000","last_became_active":"2024-03-30T13:14:06.677850+0000","last_became_peered":"2024-03-30T13:14:06.677850+0000","last_unstale":"2024-03-30T13:14:28.176274+0000","last_undegraded":"2024-03-30T13:14:28.176274+0000","last_fullsized":"2024-03-30T13:14:28.176274+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:14:13.517128+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:14:13.517128+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:35.282801+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":393,"reported_epoch":511,"state":"active+clean","last_fresh":"2024-03-30T13:14:22.153430+0000","last_change":"2024-03-30T13:14:07.271120+0000","last_active":"2024-03-30T13:14:22.153430+0000","last_peered":"2024-03-30T13:14:22.153430+0000","last_clean":"2024-03-30T13:14:22.153430+0000","last_became_active":"2024-03-30T13:14:06.679771+0000","last_became_peered":"2024-03-30T13:14:06.679771+0000","last_unstale":"2024-03-30T13:14:22.153430+0000","last_undegraded":"2024-03-30T13:14:22.153430+0000","last_fullsized":"2024-03-30T13:14:22.153430+0000","mapping_epoch":507,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":508,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:14:07.270866+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:14:07.270866+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:35.512707+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15680733,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":159,"num_bytes_recovered":387849892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":117080064,"data_stored":117080064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834174,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63056,"kb_used_data":26756,"kb_used_omap":5,"kb_used_meta":36282,"kb_avail":94308784,"statfs":{"total":96636764160,"available":96572194816,"internally_reserved":0,"allocated":27398144,"data_stored":23972294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":37152866},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":441,"seq":1894080577568,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59344,"kb_used_data":22148,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94312496,"statfs":{"total":96636764160,"available":96575995904,"internally_reserved":0,"allocated":22679552,"data_stored":19246943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9343,"internal_metadata":38067073},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509671,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50388,"kb_used_data":15048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94321452,"statfs":{"total":96636764160,"available":96585166848,"internally_reserved":0,"allocated":15409152,"data_stored":11976543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":461,"seq":1979979923476,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49484,"kb_used_data":16896,"kb_used_omap":7,"kb_used_meta":32568,"kb_avail":94322356,"statfs":{"total":96636764160,"available":96586092544,"internally_reserved":0,"allocated":17301504,"data_stored":13875654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7692,"internal_metadata":33350132},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509672,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50776,"kb_used_data":19596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321064,"statfs":{"total":96636764160,"available":96584769536,"internally_reserved":0,"allocated":20066304,"data_stored":16633695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297066,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53784,"kb_used_data":16140,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94318056,"statfs":{"total":96636764160,"available":96581689344,"internally_reserved":0,"allocated":16527360,"data_stored":13094751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147380,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52228,"kb_used_data":20536,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94319612,"statfs":{"total":96636764160,"available":96583282688,"internally_reserved":0,"allocated":21028864,"data_stored":17603014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11071488,"data_stored":11071488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9166848,"data_stored":9166848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":14798848,"data_stored":14798848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16437248,"data_stored":16437248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":21168128,"data_stored":21168128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:14:31.363 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:14:31.363 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:14:31.363 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-03-30T13:14:31.363 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-30T13:14:31.363 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-30T13:14:31.366 INFO:tasks.ceph.osd.5:Started 2024-03-30T13:14:31.366 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-30T13:14:31.430 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:14:31.430+0000 7fbe7e0997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:31.438 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:14:31.438+0000 7fbe7e0997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:31.438 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:14:31.438+0000 7fbe7e0997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:31.487 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T13:14:31.488 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T13:14:31.488 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T13:14:31.488 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T13:14:31.488 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T13:14:31.488 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T13:14:31.488 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-03-30T13:14:31.488 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-03-30T13:14:31.488 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-03-30T13:14:31.488 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-03-30T13:14:31.488 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-03-30T13:14:31.488 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T13:14:31.490 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:14:31.490 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-03-30T13:14:31.981 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:14:31.982+0000 7fbe7e0997c0 -1 Falling back to public interface 2024-03-30T13:14:35.781 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:14:35.782+0000 7fbe7e0997c0 -1 osd.5 506 log_to_monitors true 2024-03-30T13:14:36.240 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:14:36.242+0000 7fbe75e6e640 -1 osd.5 506 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:14:36.492 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-30T13:14:36.625 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T13:14:36.625 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T13:14:36.625 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T13:14:36.625 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T13:14:36.638 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:14:36.639 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-03-30T13:14:36.639 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 5 2024-03-30T13:14:36.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:36.763+0000 7febd95e3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:36.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:36.779+0000 7febd95e3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:36.964 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:14:36.959+0000 7fbb38f08640 -1 mon.a@0(leader).osd e514 definitely_dead 0 2024-03-30T13:14:37.221 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.5. 2024-03-30T13:14:37.252 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-03-30T13:14:37.252 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:14:37.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:37.375+0000 7f89137d2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:37.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:37.391+0000 7f89137d2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:38.215 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:14:38.248 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:14:38.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:38.355+0000 7f66c0edd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:38.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:38.371+0000 7f66c0edd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:39.248 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:14:39.272 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:14:39.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:39.386+0000 7ffbcb30e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:39.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:39.402+0000 7ffbcb30e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:40.268 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:14:40.295 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:14:40.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:40.410+0000 7f3cd85d3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:40.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:40.426+0000 7f3cd85d3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:41.291 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:14:41.314 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:14:41.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:41.430+0000 7f59fa5eb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:41.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:41.446+0000 7f59fa5eb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:42.296 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:14:42.322 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:14:42.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:42.438+0000 7fdf3ab23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:42.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:42.454+0000 7fdf3ab23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:43.306 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:14:43.330 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:14:43.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:43.442+0000 7f00ebc77640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:43.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:43.458+0000 7f00ebc77640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:44.317 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:14:44.341 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:14:44.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:44.458+0000 7f7d612f7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:44.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:44.474+0000 7f7d612f7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:45.333 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:14:45.356 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:14:45.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:45.474+0000 7f86dc64a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:45.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:45.490+0000 7f86dc64a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:46.354 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:14:46.379 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:14:46.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:46.494+0000 7fcf1ff36640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:46.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:46.510+0000 7fcf1ff36640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:47.344 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:14:47.374 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:14:47.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:47.494+0000 7fde8f80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:47.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:47.510+0000 7fde8f80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:48.360 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:14:48.385 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:14:48.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:48.502+0000 7efd37ef6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:48.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:48.518+0000 7efd37ef6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:49.374 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:14:49.401 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:14:49.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:49.510+0000 7feb4cb25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:49.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:49.526+0000 7feb4cb25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:50.389 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:14:50.415 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:14:50.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:50.530+0000 7f28409a7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:50.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:50.546+0000 7f28409a7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:51.407 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:14:51.434 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:14:51.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:51.550+0000 7fb89ca4e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:51.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:51.566+0000 7fb89ca4e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:52.417 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:14:52.441 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:14:52.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:52.554+0000 7f9269d40640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:52.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:52.570+0000 7f9269d40640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:53.430 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:14:53.456 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:14:53.456 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:14:53.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:53.574+0000 7fc675485640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:53.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:53.586+0000 7fc675485640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:53.763 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:14:53.763 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:14:53.781 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1104,"stamp":"2024-03-30T13:14:53.428572+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":367379263,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":428524,"kb_used_data":144752,"kb_used_omap":53,"kb_used_meta":283594,"kb_avail":754546196,"statfs":{"total":773094113280,"available":772655304704,"internally_reserved":0,"allocated":148226048,"data_stored":119929627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":54307,"internal_metadata":290401245},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.046866"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1216,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-30T13:14:49.661016+0000","last_change":"2024-03-30T13:14:38.236713+0000","last_active":"2024-03-30T13:14:49.661016+0000","last_peered":"2024-03-30T13:14:49.661016+0000","last_clean":"2024-03-30T13:14:49.661016+0000","last_became_active":"2024-03-30T13:14:38.236046+0000","last_became_peered":"2024-03-30T13:14:38.236046+0000","last_unstale":"2024-03-30T13:14:49.661016+0000","last_undegraded":"2024-03-30T13:14:49.661016+0000","last_fullsized":"2024-03-30T13:14:49.661016+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:14:07.953391+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:14:07.953391+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:22.322503+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22941225,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1127,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-03-30T13:14:48.367462+0000","last_change":"2024-03-30T13:13:42.391739+0000","last_active":"2024-03-30T13:14:48.367462+0000","last_peered":"2024-03-30T13:14:48.367462+0000","last_clean":"2024-03-30T13:14:48.367462+0000","last_became_active":"2024-03-30T13:12:00.216256+0000","last_became_peered":"2024-03-30T13:12:00.216256+0000","last_unstale":"2024-03-30T13:14:48.367462+0000","last_undegraded":"2024-03-30T13:14:48.367462+0000","last_fullsized":"2024-03-30T13:14:48.367462+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:42.391496+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_clean_scrub_stamp":"2024-03-30T13:13:42.391496+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":745,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-03-30T13:14:52.419873+0000","last_change":"2024-03-30T13:13:28.025038+0000","last_active":"2024-03-30T13:14:52.419873+0000","last_peered":"2024-03-30T13:14:52.419873+0000","last_clean":"2024-03-30T13:14:52.419873+0000","last_became_active":"2024-03-30T13:12:51.212130+0000","last_became_peered":"2024-03-30T13:12:51.212130+0000","last_unstale":"2024-03-30T13:14:52.419873+0000","last_undegraded":"2024-03-30T13:14:52.419873+0000","last_fullsized":"2024-03-30T13:14:52.419873+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:13:28.024596+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:13:28.024596+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":687,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-03-30T13:14:52.419473+0000","last_change":"2024-03-30T13:13:47.119549+0000","last_active":"2024-03-30T13:14:52.419473+0000","last_peered":"2024-03-30T13:14:52.419473+0000","last_clean":"2024-03-30T13:14:52.419473+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:14:52.419473+0000","last_undegraded":"2024-03-30T13:14:52.419473+0000","last_fullsized":"2024-03-30T13:14:52.419473+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:13:47.119125+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:13:47.119125+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:15.963858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":61,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-03-30T13:14:51.557042+0000","last_change":"2024-03-30T13:14:38.234542+0000","last_active":"2024-03-30T13:14:51.557042+0000","last_peered":"2024-03-30T13:14:51.557042+0000","last_clean":"2024-03-30T13:14:51.557042+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:14:51.557042+0000","last_undegraded":"2024-03-30T13:14:51.557042+0000","last_fullsized":"2024-03-30T13:14:51.557042+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:49.332375+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:13:49.332375+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:19.045572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1225,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-03-30T13:14:51.554418+0000","last_change":"2024-03-30T13:14:38.238826+0000","last_active":"2024-03-30T13:14:51.554418+0000","last_peered":"2024-03-30T13:14:51.554418+0000","last_clean":"2024-03-30T13:14:51.554418+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:14:51.554418+0000","last_undegraded":"2024-03-30T13:14:51.554418+0000","last_fullsized":"2024-03-30T13:14:51.554418+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:13:35.802156+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:13:35.802156+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:54.127627+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":730,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-03-30T13:14:52.419544+0000","last_change":"2024-03-30T13:14:26.237658+0000","last_active":"2024-03-30T13:14:52.419544+0000","last_peered":"2024-03-30T13:14:52.419544+0000","last_clean":"2024-03-30T13:14:52.419544+0000","last_became_active":"2024-03-30T13:12:00.202949+0000","last_became_peered":"2024-03-30T13:12:00.202949+0000","last_unstale":"2024-03-30T13:14:52.419544+0000","last_undegraded":"2024-03-30T13:14:52.419544+0000","last_fullsized":"2024-03-30T13:14:52.419544+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:14:26.237251+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:14:26.237251+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:30.208212+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":488,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-30T13:14:49.383655+0000","last_change":"2024-03-30T13:14:39.618860+0000","last_active":"2024-03-30T13:14:49.383655+0000","last_peered":"2024-03-30T13:14:49.383655+0000","last_clean":"2024-03-30T13:14:49.383655+0000","last_became_active":"2024-03-30T13:14:38.233325+0000","last_became_peered":"2024-03-30T13:14:38.233325+0000","last_unstale":"2024-03-30T13:14:49.383655+0000","last_undegraded":"2024-03-30T13:14:49.383655+0000","last_fullsized":"2024-03-30T13:14:49.383655+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:14:39.618688+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:14:39.618688+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:00.662395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1333,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-03-30T13:14:51.554336+0000","last_change":"2024-03-30T13:14:38.238691+0000","last_active":"2024-03-30T13:14:51.554336+0000","last_peered":"2024-03-30T13:14:51.554336+0000","last_clean":"2024-03-30T13:14:51.554336+0000","last_became_active":"2024-03-30T13:14:38.238174+0000","last_became_peered":"2024-03-30T13:14:38.238174+0000","last_unstale":"2024-03-30T13:14:51.554336+0000","last_undegraded":"2024-03-30T13:14:51.554336+0000","last_fullsized":"2024-03-30T13:14:51.554336+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:14:24.557218+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:14:24.557218+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:30.601618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":71059072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1001,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-03-30T13:14:48.367380+0000","last_change":"2024-03-30T13:13:38.374659+0000","last_active":"2024-03-30T13:14:48.367380+0000","last_peered":"2024-03-30T13:14:48.367380+0000","last_clean":"2024-03-30T13:14:48.367380+0000","last_became_active":"2024-03-30T13:12:00.203605+0000","last_became_peered":"2024-03-30T13:12:00.203605+0000","last_unstale":"2024-03-30T13:14:48.367380+0000","last_undegraded":"2024-03-30T13:14:48.367380+0000","last_fullsized":"2024-03-30T13:14:48.367380+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:38.374502+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:13:38.374502+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1188,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-30T13:14:49.660772+0000","last_change":"2024-03-30T13:13:57.067928+0000","last_active":"2024-03-30T13:14:49.660772+0000","last_peered":"2024-03-30T13:14:49.660772+0000","last_clean":"2024-03-30T13:14:49.660772+0000","last_became_active":"2024-03-30T13:12:00.202952+0000","last_became_peered":"2024-03-30T13:12:00.202952+0000","last_unstale":"2024-03-30T13:14:49.660772+0000","last_undegraded":"2024-03-30T13:14:49.660772+0000","last_fullsized":"2024-03-30T13:14:49.660772+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:13:57.067688+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:13:57.067688+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:13.208052+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1306,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-30T13:14:49.660860+0000","last_change":"2024-03-30T13:14:38.238033+0000","last_active":"2024-03-30T13:14:49.660860+0000","last_peered":"2024-03-30T13:14:49.660860+0000","last_clean":"2024-03-30T13:14:49.660860+0000","last_became_active":"2024-03-30T13:14:38.236424+0000","last_became_peered":"2024-03-30T13:14:38.236424+0000","last_unstale":"2024-03-30T13:14:49.660860+0000","last_undegraded":"2024-03-30T13:14:49.660860+0000","last_fullsized":"2024-03-30T13:14:49.660860+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:13:48.913862+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:13:48.913862+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:10.572041+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":76975004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":74,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-03-30T13:14:51.557018+0000","last_change":"2024-03-30T13:14:38.236713+0000","last_active":"2024-03-30T13:14:51.557018+0000","last_peered":"2024-03-30T13:14:51.557018+0000","last_clean":"2024-03-30T13:14:51.557018+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:14:51.557018+0000","last_undegraded":"2024-03-30T13:14:51.557018+0000","last_fullsized":"2024-03-30T13:14:51.557018+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:13:52.331764+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:13:52.331764+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:10.326899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":666,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-03-30T13:14:52.419473+0000","last_change":"2024-03-30T13:14:47.424410+0000","last_active":"2024-03-30T13:14:52.419473+0000","last_peered":"2024-03-30T13:14:52.419473+0000","last_clean":"2024-03-30T13:14:52.419473+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:14:52.419473+0000","last_undegraded":"2024-03-30T13:14:52.419473+0000","last_fullsized":"2024-03-30T13:14:52.419473+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:14:47.424173+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:14:47.424173+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:47.850501+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":58,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-03-30T13:14:51.557085+0000","last_change":"2024-03-30T13:14:38.236720+0000","last_active":"2024-03-30T13:14:51.557085+0000","last_peered":"2024-03-30T13:14:51.557085+0000","last_clean":"2024-03-30T13:14:51.557085+0000","last_became_active":"2024-03-30T13:14:38.235512+0000","last_became_peered":"2024-03-30T13:14:38.235512+0000","last_unstale":"2024-03-30T13:14:51.557085+0000","last_undegraded":"2024-03-30T13:14:51.557085+0000","last_fullsized":"2024-03-30T13:14:51.557085+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:14:13.517128+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:14:13.517128+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:36.511883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":431,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-30T13:14:49.383780+0000","last_change":"2024-03-30T13:14:38.231723+0000","last_active":"2024-03-30T13:14:49.383780+0000","last_peered":"2024-03-30T13:14:49.383780+0000","last_clean":"2024-03-30T13:14:49.383780+0000","last_became_active":"2024-03-30T13:14:38.231065+0000","last_became_peered":"2024-03-30T13:14:38.231065+0000","last_unstale":"2024-03-30T13:14:49.383780+0000","last_undegraded":"2024-03-30T13:14:49.383780+0000","last_fullsized":"2024-03-30T13:14:49.383780+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:14:07.270866+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:14:07.270866+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:09.462813+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15680733,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":367379263,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834179,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57692,"kb_used_data":21384,"kb_used_omap":5,"kb_used_meta":36282,"kb_avail":94314148,"statfs":{"total":96636764160,"available":96577687552,"internally_reserved":0,"allocated":21897216,"data_stored":18364414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":37152866},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157445,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63796,"kb_used_data":26152,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94308044,"statfs":{"total":96636764160,"available":96571437056,"internally_reserved":0,"allocated":26779648,"data_stored":23233244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":441,"seq":1894080577572,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55024,"kb_used_data":17820,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94316816,"statfs":{"total":96636764160,"available":96580419584,"internally_reserved":0,"allocated":18247680,"data_stored":14721679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9343,"internal_metadata":38067073},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509675,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47944,"kb_used_data":12600,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323896,"statfs":{"total":96636764160,"available":96587669504,"internally_reserved":0,"allocated":12902400,"data_stored":9369598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":461,"seq":1979979923481,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48624,"kb_used_data":11876,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94323216,"statfs":{"total":96636764160,"available":96586973184,"internally_reserved":0,"allocated":12161024,"data_stored":8628222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7692,"internal_metadata":37609972},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509676,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50944,"kb_used_data":19764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320896,"statfs":{"total":96636764160,"available":96584597504,"internally_reserved":0,"allocated":20238336,"data_stored":16712335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297071,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54000,"kb_used_data":16356,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94317840,"statfs":{"total":96636764160,"available":96581468160,"internally_reserved":0,"allocated":16748544,"data_stored":13195339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147385,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50500,"kb_used_data":18800,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321340,"statfs":{"total":96636764160,"available":96585052160,"internally_reserved":0,"allocated":19251200,"data_stored":15704796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:14:53.782 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:14:53.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:53.902+0000 7f2249d23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:53.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:53.914+0000 7f2249d23640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:54.082 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:14:54.082 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:14:54.098 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1104,"stamp":"2024-03-30T13:14:53.428572+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":367379263,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":428524,"kb_used_data":144752,"kb_used_omap":53,"kb_used_meta":283594,"kb_avail":754546196,"statfs":{"total":773094113280,"available":772655304704,"internally_reserved":0,"allocated":148226048,"data_stored":119929627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":54307,"internal_metadata":290401245},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.046866"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1216,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-30T13:14:49.661016+0000","last_change":"2024-03-30T13:14:38.236713+0000","last_active":"2024-03-30T13:14:49.661016+0000","last_peered":"2024-03-30T13:14:49.661016+0000","last_clean":"2024-03-30T13:14:49.661016+0000","last_became_active":"2024-03-30T13:14:38.236046+0000","last_became_peered":"2024-03-30T13:14:38.236046+0000","last_unstale":"2024-03-30T13:14:49.661016+0000","last_undegraded":"2024-03-30T13:14:49.661016+0000","last_fullsized":"2024-03-30T13:14:49.661016+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:14:07.953391+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:14:07.953391+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:22.322503+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22941225,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1127,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-03-30T13:14:48.367462+0000","last_change":"2024-03-30T13:13:42.391739+0000","last_active":"2024-03-30T13:14:48.367462+0000","last_peered":"2024-03-30T13:14:48.367462+0000","last_clean":"2024-03-30T13:14:48.367462+0000","last_became_active":"2024-03-30T13:12:00.216256+0000","last_became_peered":"2024-03-30T13:12:00.216256+0000","last_unstale":"2024-03-30T13:14:48.367462+0000","last_undegraded":"2024-03-30T13:14:48.367462+0000","last_fullsized":"2024-03-30T13:14:48.367462+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:42.391496+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_clean_scrub_stamp":"2024-03-30T13:13:42.391496+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":745,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-03-30T13:14:52.419873+0000","last_change":"2024-03-30T13:13:28.025038+0000","last_active":"2024-03-30T13:14:52.419873+0000","last_peered":"2024-03-30T13:14:52.419873+0000","last_clean":"2024-03-30T13:14:52.419873+0000","last_became_active":"2024-03-30T13:12:51.212130+0000","last_became_peered":"2024-03-30T13:12:51.212130+0000","last_unstale":"2024-03-30T13:14:52.419873+0000","last_undegraded":"2024-03-30T13:14:52.419873+0000","last_fullsized":"2024-03-30T13:14:52.419873+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:13:28.024596+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:13:28.024596+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":687,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-03-30T13:14:52.419473+0000","last_change":"2024-03-30T13:13:47.119549+0000","last_active":"2024-03-30T13:14:52.419473+0000","last_peered":"2024-03-30T13:14:52.419473+0000","last_clean":"2024-03-30T13:14:52.419473+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:14:52.419473+0000","last_undegraded":"2024-03-30T13:14:52.419473+0000","last_fullsized":"2024-03-30T13:14:52.419473+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:13:47.119125+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:13:47.119125+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:15.963858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":61,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-03-30T13:14:51.557042+0000","last_change":"2024-03-30T13:14:38.234542+0000","last_active":"2024-03-30T13:14:51.557042+0000","last_peered":"2024-03-30T13:14:51.557042+0000","last_clean":"2024-03-30T13:14:51.557042+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:14:51.557042+0000","last_undegraded":"2024-03-30T13:14:51.557042+0000","last_fullsized":"2024-03-30T13:14:51.557042+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:49.332375+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:13:49.332375+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:19.045572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1225,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-03-30T13:14:51.554418+0000","last_change":"2024-03-30T13:14:38.238826+0000","last_active":"2024-03-30T13:14:51.554418+0000","last_peered":"2024-03-30T13:14:51.554418+0000","last_clean":"2024-03-30T13:14:51.554418+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:14:51.554418+0000","last_undegraded":"2024-03-30T13:14:51.554418+0000","last_fullsized":"2024-03-30T13:14:51.554418+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:13:35.802156+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:13:35.802156+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:54.127627+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":730,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-03-30T13:14:52.419544+0000","last_change":"2024-03-30T13:14:26.237658+0000","last_active":"2024-03-30T13:14:52.419544+0000","last_peered":"2024-03-30T13:14:52.419544+0000","last_clean":"2024-03-30T13:14:52.419544+0000","last_became_active":"2024-03-30T13:12:00.202949+0000","last_became_peered":"2024-03-30T13:12:00.202949+0000","last_unstale":"2024-03-30T13:14:52.419544+0000","last_undegraded":"2024-03-30T13:14:52.419544+0000","last_fullsized":"2024-03-30T13:14:52.419544+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:14:26.237251+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:14:26.237251+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:30.208212+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":488,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-30T13:14:49.383655+0000","last_change":"2024-03-30T13:14:39.618860+0000","last_active":"2024-03-30T13:14:49.383655+0000","last_peered":"2024-03-30T13:14:49.383655+0000","last_clean":"2024-03-30T13:14:49.383655+0000","last_became_active":"2024-03-30T13:14:38.233325+0000","last_became_peered":"2024-03-30T13:14:38.233325+0000","last_unstale":"2024-03-30T13:14:49.383655+0000","last_undegraded":"2024-03-30T13:14:49.383655+0000","last_fullsized":"2024-03-30T13:14:49.383655+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:14:39.618688+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:14:39.618688+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:00.662395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1333,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-03-30T13:14:51.554336+0000","last_change":"2024-03-30T13:14:38.238691+0000","last_active":"2024-03-30T13:14:51.554336+0000","last_peered":"2024-03-30T13:14:51.554336+0000","last_clean":"2024-03-30T13:14:51.554336+0000","last_became_active":"2024-03-30T13:14:38.238174+0000","last_became_peered":"2024-03-30T13:14:38.238174+0000","last_unstale":"2024-03-30T13:14:51.554336+0000","last_undegraded":"2024-03-30T13:14:51.554336+0000","last_fullsized":"2024-03-30T13:14:51.554336+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:14:24.557218+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:14:24.557218+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:30.601618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":71059072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1001,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-03-30T13:14:48.367380+0000","last_change":"2024-03-30T13:13:38.374659+0000","last_active":"2024-03-30T13:14:48.367380+0000","last_peered":"2024-03-30T13:14:48.367380+0000","last_clean":"2024-03-30T13:14:48.367380+0000","last_became_active":"2024-03-30T13:12:00.203605+0000","last_became_peered":"2024-03-30T13:12:00.203605+0000","last_unstale":"2024-03-30T13:14:48.367380+0000","last_undegraded":"2024-03-30T13:14:48.367380+0000","last_fullsized":"2024-03-30T13:14:48.367380+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:38.374502+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:13:38.374502+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1188,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-30T13:14:49.660772+0000","last_change":"2024-03-30T13:13:57.067928+0000","last_active":"2024-03-30T13:14:49.660772+0000","last_peered":"2024-03-30T13:14:49.660772+0000","last_clean":"2024-03-30T13:14:49.660772+0000","last_became_active":"2024-03-30T13:12:00.202952+0000","last_became_peered":"2024-03-30T13:12:00.202952+0000","last_unstale":"2024-03-30T13:14:49.660772+0000","last_undegraded":"2024-03-30T13:14:49.660772+0000","last_fullsized":"2024-03-30T13:14:49.660772+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:13:57.067688+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:13:57.067688+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:13.208052+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1306,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-30T13:14:49.660860+0000","last_change":"2024-03-30T13:14:38.238033+0000","last_active":"2024-03-30T13:14:49.660860+0000","last_peered":"2024-03-30T13:14:49.660860+0000","last_clean":"2024-03-30T13:14:49.660860+0000","last_became_active":"2024-03-30T13:14:38.236424+0000","last_became_peered":"2024-03-30T13:14:38.236424+0000","last_unstale":"2024-03-30T13:14:49.660860+0000","last_undegraded":"2024-03-30T13:14:49.660860+0000","last_fullsized":"2024-03-30T13:14:49.660860+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:13:48.913862+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:13:48.913862+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:10.572041+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":76975004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":74,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-03-30T13:14:51.557018+0000","last_change":"2024-03-30T13:14:38.236713+0000","last_active":"2024-03-30T13:14:51.557018+0000","last_peered":"2024-03-30T13:14:51.557018+0000","last_clean":"2024-03-30T13:14:51.557018+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:14:51.557018+0000","last_undegraded":"2024-03-30T13:14:51.557018+0000","last_fullsized":"2024-03-30T13:14:51.557018+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:13:52.331764+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:13:52.331764+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:10.326899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":666,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-03-30T13:14:52.419473+0000","last_change":"2024-03-30T13:14:47.424410+0000","last_active":"2024-03-30T13:14:52.419473+0000","last_peered":"2024-03-30T13:14:52.419473+0000","last_clean":"2024-03-30T13:14:52.419473+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:14:52.419473+0000","last_undegraded":"2024-03-30T13:14:52.419473+0000","last_fullsized":"2024-03-30T13:14:52.419473+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:14:47.424173+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:14:47.424173+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:47.850501+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":58,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-03-30T13:14:51.557085+0000","last_change":"2024-03-30T13:14:38.236720+0000","last_active":"2024-03-30T13:14:51.557085+0000","last_peered":"2024-03-30T13:14:51.557085+0000","last_clean":"2024-03-30T13:14:51.557085+0000","last_became_active":"2024-03-30T13:14:38.235512+0000","last_became_peered":"2024-03-30T13:14:38.235512+0000","last_unstale":"2024-03-30T13:14:51.557085+0000","last_undegraded":"2024-03-30T13:14:51.557085+0000","last_fullsized":"2024-03-30T13:14:51.557085+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:14:13.517128+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:14:13.517128+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:36.511883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":431,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-30T13:14:49.383780+0000","last_change":"2024-03-30T13:14:38.231723+0000","last_active":"2024-03-30T13:14:49.383780+0000","last_peered":"2024-03-30T13:14:49.383780+0000","last_clean":"2024-03-30T13:14:49.383780+0000","last_became_active":"2024-03-30T13:14:38.231065+0000","last_became_peered":"2024-03-30T13:14:38.231065+0000","last_unstale":"2024-03-30T13:14:49.383780+0000","last_undegraded":"2024-03-30T13:14:49.383780+0000","last_fullsized":"2024-03-30T13:14:49.383780+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:14:07.270866+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:14:07.270866+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:09.462813+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15680733,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":367379263,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834179,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57692,"kb_used_data":21384,"kb_used_omap":5,"kb_used_meta":36282,"kb_avail":94314148,"statfs":{"total":96636764160,"available":96577687552,"internally_reserved":0,"allocated":21897216,"data_stored":18364414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":37152866},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157445,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63796,"kb_used_data":26152,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94308044,"statfs":{"total":96636764160,"available":96571437056,"internally_reserved":0,"allocated":26779648,"data_stored":23233244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":441,"seq":1894080577572,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55024,"kb_used_data":17820,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94316816,"statfs":{"total":96636764160,"available":96580419584,"internally_reserved":0,"allocated":18247680,"data_stored":14721679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9343,"internal_metadata":38067073},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509675,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47944,"kb_used_data":12600,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323896,"statfs":{"total":96636764160,"available":96587669504,"internally_reserved":0,"allocated":12902400,"data_stored":9369598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":461,"seq":1979979923481,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48624,"kb_used_data":11876,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94323216,"statfs":{"total":96636764160,"available":96586973184,"internally_reserved":0,"allocated":12161024,"data_stored":8628222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7692,"internal_metadata":37609972},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509676,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50944,"kb_used_data":19764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320896,"statfs":{"total":96636764160,"available":96584597504,"internally_reserved":0,"allocated":20238336,"data_stored":16712335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297071,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54000,"kb_used_data":16356,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94317840,"statfs":{"total":96636764160,"available":96581468160,"internally_reserved":0,"allocated":16748544,"data_stored":13195339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147385,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50500,"kb_used_data":18800,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321340,"statfs":{"total":96636764160,"available":96585052160,"internally_reserved":0,"allocated":19251200,"data_stored":15704796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:14:54.100 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:14:54.100 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:14:54.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:54.214+0000 7f3a40d01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:54.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:54.226+0000 7f3a40d01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:54.400 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:14:54.400 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":531,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:14:53.420298+0000","last_up_change":"2024-03-30T13:14:37.211591+0000","last_in_change":"2024-03-30T13:14:37.211591+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":436,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":346,"last_clean_end":457,"up_from":461,"up_thru":515,"down_at":458,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2034332759},{"type":"v1","addr":"172.21.15.67:6818","nonce":2034332759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2034332759},{"type":"v1","addr":"172.21.15.67:6821","nonce":2034332759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2034332759},{"type":"v1","addr":"172.21.15.67:6828","nonce":2034332759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2034332759},{"type":"v1","addr":"172.21.15.67:6825","nonce":2034332759}]},"public_addr":"172.21.15.67:6818/2034332759","cluster_addr":"172.21.15.67:6821/2034332759","heartbeat_back_addr":"172.21.15.67:6828/2034332759","heartbeat_front_addr":"172.21.15.67:6825/2034332759","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":479,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":388,"up_from":393,"up_thru":515,"down_at":389,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6801","nonce":3837194718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6803","nonce":3837194718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6807","nonce":3837194718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6805","nonce":3837194718}]},"public_addr":"172.21.15.67:6801/3837194718","cluster_addr":"172.21.15.67:6803/3837194718","heartbeat_back_addr":"172.21.15.67:6807/3837194718","heartbeat_front_addr":"172.21.15.67:6805/3837194718","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":250,"last_clean_end":435,"up_from":441,"up_thru":507,"down_at":436,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3152653542},{"type":"v1","addr":"172.21.15.82:6803","nonce":3152653542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":3152653542},{"type":"v1","addr":"172.21.15.82:6807","nonce":3152653542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":3152653542},{"type":"v1","addr":"172.21.15.82:6815","nonce":3152653542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":3152653542},{"type":"v1","addr":"172.21.15.82:6811","nonce":3152653542}]},"public_addr":"172.21.15.82:6803/3152653542","cluster_addr":"172.21.15.82:6807/3152653542","heartbeat_back_addr":"172.21.15.82:6815/3152653542","heartbeat_front_addr":"172.21.15.82:6811/3152653542","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":488,"last_clean_end":506,"up_from":515,"up_thru":515,"down_at":507,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6817","nonce":292505453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6819","nonce":292505453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6823","nonce":292505453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6821","nonce":292505453}]},"public_addr":"172.21.15.82:6817/292505453","cluster_addr":"172.21.15.82:6819/292505453","heartbeat_back_addr":"172.21.15.82:6823/292505453","heartbeat_front_addr":"172.21.15.82:6821/292505453","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":227,"last_clean_end":363,"up_from":372,"up_thru":515,"down_at":364,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6801","nonce":189641108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6806","nonce":189641108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6814","nonce":189641108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6810","nonce":189641108}]},"public_addr":"172.21.15.82:6801/189641108","cluster_addr":"172.21.15.82:6806/189641108","heartbeat_back_addr":"172.21.15.82:6814/189641108","heartbeat_front_addr":"172.21.15.82:6810/189641108","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":298,"last_clean_end":409,"up_from":419,"up_thru":461,"down_at":410,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6825","nonce":2166599911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6827","nonce":2166599911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6831","nonce":2166599911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6829","nonce":2166599911}]},"public_addr":"172.21.15.82:6825/2166599911","cluster_addr":"172.21.15.82:6827/2166599911","heartbeat_back_addr":"172.21.15.82:6831/2166599911","heartbeat_front_addr":"172.21.15.82:6829/2166599911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:14:45.583060+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:14:17.808366+0000","dead_epoch":458},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:14:12.062873+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:14:41.722471+0000","dead_epoch":389},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:14:27.768387+0000","dead_epoch":436},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:14:46.777563+0000","dead_epoch":507},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:14:21.289223+0000","dead_epoch":364},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:14:29.816448+0000","dead_epoch":410}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:14:54.420 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:14:54.420 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:14:54.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:54.530+0000 7f7e36afc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:54.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:54.546+0000 7f7e36afc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:54.718 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:14:54.737 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:14:54.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:54.858+0000 7fa225d53640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:54.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:54.874+0000 7fa225d53640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:55.055 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:14:55.055 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:14:55.075 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:14:55.075 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:14:55.075 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:14:55.075 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:14:55.075 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 6, 3, 7, 4, 1, 5] 2024-03-30T13:14:55.075 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:14:55.075 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:14:55.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:55.182+0000 7f7c27db9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:55.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:55.198+0000 7f7c27db9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:55.370 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:14:55.371 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:14:55.390 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1105,"stamp":"2024-03-30T13:14:54.117402+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":367379263,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":428524,"kb_used_data":144752,"kb_used_omap":53,"kb_used_meta":283594,"kb_avail":754546196,"statfs":{"total":773094113280,"available":772655304704,"internally_reserved":0,"allocated":148226048,"data_stored":119929627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":54307,"internal_metadata":290401245},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001017"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1216,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-30T13:14:49.661016+0000","last_change":"2024-03-30T13:14:38.236713+0000","last_active":"2024-03-30T13:14:49.661016+0000","last_peered":"2024-03-30T13:14:49.661016+0000","last_clean":"2024-03-30T13:14:49.661016+0000","last_became_active":"2024-03-30T13:14:38.236046+0000","last_became_peered":"2024-03-30T13:14:38.236046+0000","last_unstale":"2024-03-30T13:14:49.661016+0000","last_undegraded":"2024-03-30T13:14:49.661016+0000","last_fullsized":"2024-03-30T13:14:49.661016+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:14:07.953391+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:14:07.953391+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:22.322503+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22941225,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1127,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-03-30T13:14:48.367462+0000","last_change":"2024-03-30T13:13:42.391739+0000","last_active":"2024-03-30T13:14:48.367462+0000","last_peered":"2024-03-30T13:14:48.367462+0000","last_clean":"2024-03-30T13:14:48.367462+0000","last_became_active":"2024-03-30T13:12:00.216256+0000","last_became_peered":"2024-03-30T13:12:00.216256+0000","last_unstale":"2024-03-30T13:14:48.367462+0000","last_undegraded":"2024-03-30T13:14:48.367462+0000","last_fullsized":"2024-03-30T13:14:48.367462+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:42.391496+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:10:44.578038+0000","last_clean_scrub_stamp":"2024-03-30T13:13:42.391496+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":745,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-03-30T13:14:52.419873+0000","last_change":"2024-03-30T13:13:28.025038+0000","last_active":"2024-03-30T13:14:52.419873+0000","last_peered":"2024-03-30T13:14:52.419873+0000","last_clean":"2024-03-30T13:14:52.419873+0000","last_became_active":"2024-03-30T13:12:51.212130+0000","last_became_peered":"2024-03-30T13:12:51.212130+0000","last_unstale":"2024-03-30T13:14:52.419873+0000","last_undegraded":"2024-03-30T13:14:52.419873+0000","last_fullsized":"2024-03-30T13:14:52.419873+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:13:28.024596+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:13:28.024596+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":687,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-03-30T13:14:52.419473+0000","last_change":"2024-03-30T13:13:47.119549+0000","last_active":"2024-03-30T13:14:52.419473+0000","last_peered":"2024-03-30T13:14:52.419473+0000","last_clean":"2024-03-30T13:14:52.419473+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:14:52.419473+0000","last_undegraded":"2024-03-30T13:14:52.419473+0000","last_fullsized":"2024-03-30T13:14:52.419473+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:13:47.119125+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:13:47.119125+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:15.963858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":61,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-03-30T13:14:51.557042+0000","last_change":"2024-03-30T13:14:38.234542+0000","last_active":"2024-03-30T13:14:51.557042+0000","last_peered":"2024-03-30T13:14:51.557042+0000","last_clean":"2024-03-30T13:14:51.557042+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:14:51.557042+0000","last_undegraded":"2024-03-30T13:14:51.557042+0000","last_fullsized":"2024-03-30T13:14:51.557042+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:49.332375+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:13:49.332375+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:19.045572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1225,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-03-30T13:14:51.554418+0000","last_change":"2024-03-30T13:14:38.238826+0000","last_active":"2024-03-30T13:14:51.554418+0000","last_peered":"2024-03-30T13:14:51.554418+0000","last_clean":"2024-03-30T13:14:51.554418+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:14:51.554418+0000","last_undegraded":"2024-03-30T13:14:51.554418+0000","last_fullsized":"2024-03-30T13:14:51.554418+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:13:35.802156+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:13:35.802156+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:54.127627+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":730,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-03-30T13:14:52.419544+0000","last_change":"2024-03-30T13:14:26.237658+0000","last_active":"2024-03-30T13:14:52.419544+0000","last_peered":"2024-03-30T13:14:52.419544+0000","last_clean":"2024-03-30T13:14:52.419544+0000","last_became_active":"2024-03-30T13:12:00.202949+0000","last_became_peered":"2024-03-30T13:12:00.202949+0000","last_unstale":"2024-03-30T13:14:52.419544+0000","last_undegraded":"2024-03-30T13:14:52.419544+0000","last_fullsized":"2024-03-30T13:14:52.419544+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:14:26.237251+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:14:26.237251+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:30.208212+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":488,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-30T13:14:49.383655+0000","last_change":"2024-03-30T13:14:39.618860+0000","last_active":"2024-03-30T13:14:49.383655+0000","last_peered":"2024-03-30T13:14:49.383655+0000","last_clean":"2024-03-30T13:14:49.383655+0000","last_became_active":"2024-03-30T13:14:38.233325+0000","last_became_peered":"2024-03-30T13:14:38.233325+0000","last_unstale":"2024-03-30T13:14:49.383655+0000","last_undegraded":"2024-03-30T13:14:49.383655+0000","last_fullsized":"2024-03-30T13:14:49.383655+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:14:39.618688+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:14:39.618688+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:00.662395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1333,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-03-30T13:14:51.554336+0000","last_change":"2024-03-30T13:14:38.238691+0000","last_active":"2024-03-30T13:14:51.554336+0000","last_peered":"2024-03-30T13:14:51.554336+0000","last_clean":"2024-03-30T13:14:51.554336+0000","last_became_active":"2024-03-30T13:14:38.238174+0000","last_became_peered":"2024-03-30T13:14:38.238174+0000","last_unstale":"2024-03-30T13:14:51.554336+0000","last_undegraded":"2024-03-30T13:14:51.554336+0000","last_fullsized":"2024-03-30T13:14:51.554336+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:14:24.557218+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:14:24.557218+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:30.601618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":71059072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1001,"reported_epoch":526,"state":"active+clean","last_fresh":"2024-03-30T13:14:48.367380+0000","last_change":"2024-03-30T13:13:38.374659+0000","last_active":"2024-03-30T13:14:48.367380+0000","last_peered":"2024-03-30T13:14:48.367380+0000","last_clean":"2024-03-30T13:14:48.367380+0000","last_became_active":"2024-03-30T13:12:00.203605+0000","last_became_peered":"2024-03-30T13:12:00.203605+0000","last_unstale":"2024-03-30T13:14:48.367380+0000","last_undegraded":"2024-03-30T13:14:48.367380+0000","last_fullsized":"2024-03-30T13:14:48.367380+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:38.374502+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:42.816452+0000","last_clean_scrub_stamp":"2024-03-30T13:13:38.374502+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1188,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-30T13:14:49.660772+0000","last_change":"2024-03-30T13:13:57.067928+0000","last_active":"2024-03-30T13:14:49.660772+0000","last_peered":"2024-03-30T13:14:49.660772+0000","last_clean":"2024-03-30T13:14:49.660772+0000","last_became_active":"2024-03-30T13:12:00.202952+0000","last_became_peered":"2024-03-30T13:12:00.202952+0000","last_unstale":"2024-03-30T13:14:49.660772+0000","last_undegraded":"2024-03-30T13:14:49.660772+0000","last_fullsized":"2024-03-30T13:14:49.660772+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:13:57.067688+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:13:57.067688+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:13.208052+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1306,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-30T13:14:49.660860+0000","last_change":"2024-03-30T13:14:38.238033+0000","last_active":"2024-03-30T13:14:49.660860+0000","last_peered":"2024-03-30T13:14:49.660860+0000","last_clean":"2024-03-30T13:14:49.660860+0000","last_became_active":"2024-03-30T13:14:38.236424+0000","last_became_peered":"2024-03-30T13:14:38.236424+0000","last_unstale":"2024-03-30T13:14:49.660860+0000","last_undegraded":"2024-03-30T13:14:49.660860+0000","last_fullsized":"2024-03-30T13:14:49.660860+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:13:48.913862+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:13:48.913862+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:10.572041+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":76975004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":74,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-03-30T13:14:51.557018+0000","last_change":"2024-03-30T13:14:38.236713+0000","last_active":"2024-03-30T13:14:51.557018+0000","last_peered":"2024-03-30T13:14:51.557018+0000","last_clean":"2024-03-30T13:14:51.557018+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:14:51.557018+0000","last_undegraded":"2024-03-30T13:14:51.557018+0000","last_fullsized":"2024-03-30T13:14:51.557018+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:13:52.331764+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:13:52.331764+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:10.326899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":666,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-03-30T13:14:52.419473+0000","last_change":"2024-03-30T13:14:47.424410+0000","last_active":"2024-03-30T13:14:52.419473+0000","last_peered":"2024-03-30T13:14:52.419473+0000","last_clean":"2024-03-30T13:14:52.419473+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:14:52.419473+0000","last_undegraded":"2024-03-30T13:14:52.419473+0000","last_fullsized":"2024-03-30T13:14:52.419473+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:14:47.424173+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:14:47.424173+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:47.850501+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":58,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-03-30T13:14:51.557085+0000","last_change":"2024-03-30T13:14:38.236720+0000","last_active":"2024-03-30T13:14:51.557085+0000","last_peered":"2024-03-30T13:14:51.557085+0000","last_clean":"2024-03-30T13:14:51.557085+0000","last_became_active":"2024-03-30T13:14:38.235512+0000","last_became_peered":"2024-03-30T13:14:38.235512+0000","last_unstale":"2024-03-30T13:14:51.557085+0000","last_undegraded":"2024-03-30T13:14:51.557085+0000","last_fullsized":"2024-03-30T13:14:51.557085+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:14:13.517128+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:14:13.517128+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:36.511883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":431,"reported_epoch":527,"state":"active+clean","last_fresh":"2024-03-30T13:14:49.383780+0000","last_change":"2024-03-30T13:14:38.231723+0000","last_active":"2024-03-30T13:14:49.383780+0000","last_peered":"2024-03-30T13:14:49.383780+0000","last_clean":"2024-03-30T13:14:49.383780+0000","last_became_active":"2024-03-30T13:14:38.231065+0000","last_became_peered":"2024-03-30T13:14:38.231065+0000","last_unstale":"2024-03-30T13:14:49.383780+0000","last_undegraded":"2024-03-30T13:14:49.383780+0000","last_fullsized":"2024-03-30T13:14:49.383780+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:14:07.270866+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:14:07.270866+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:09.462813+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15680733,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":367379263,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834179,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57692,"kb_used_data":21384,"kb_used_omap":5,"kb_used_meta":36282,"kb_avail":94314148,"statfs":{"total":96636764160,"available":96577687552,"internally_reserved":0,"allocated":21897216,"data_stored":18364414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":37152866},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157445,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63796,"kb_used_data":26152,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94308044,"statfs":{"total":96636764160,"available":96571437056,"internally_reserved":0,"allocated":26779648,"data_stored":23233244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":441,"seq":1894080577572,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55024,"kb_used_data":17820,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94316816,"statfs":{"total":96636764160,"available":96580419584,"internally_reserved":0,"allocated":18247680,"data_stored":14721679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9343,"internal_metadata":38067073},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509675,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47944,"kb_used_data":12600,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323896,"statfs":{"total":96636764160,"available":96587669504,"internally_reserved":0,"allocated":12902400,"data_stored":9369598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":461,"seq":1979979923481,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48624,"kb_used_data":11876,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94323216,"statfs":{"total":96636764160,"available":96586973184,"internally_reserved":0,"allocated":12161024,"data_stored":8628222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7692,"internal_metadata":37609972},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509676,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50944,"kb_used_data":19764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320896,"statfs":{"total":96636764160,"available":96584597504,"internally_reserved":0,"allocated":20238336,"data_stored":16712335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297071,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54000,"kb_used_data":16356,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94317840,"statfs":{"total":96636764160,"available":96581468160,"internally_reserved":0,"allocated":16748544,"data_stored":13195339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147385,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50500,"kb_used_data":18800,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321340,"statfs":{"total":96636764160,"available":96585052160,"internally_reserved":0,"allocated":19251200,"data_stored":15704796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:14:55.391 INFO:tasks.thrashosds.thrasher:Choosing PG 1.5 with acting set [4, 2, 6] 2024-03-30T13:14:55.391 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [0, 2, 6, 3, 7, 4, 1, 5] 2024-03-30T13:14:55.391 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-30T13:14:55.391 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:14:55.458 INFO:tasks.ceph.osd.4:Stopped 2024-03-30T13:14:55.458 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 2, 6, 3, 7, 4, 1, 5] 2024-03-30T13:14:55.459 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 4 2024-03-30T13:14:55.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:55.578+0000 7fb4a8b09640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:55.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:55.594+0000 7fb4a8b09640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:55.777 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:14:55.774+0000 7fbb38f08640 -1 mon.a@0(leader).osd e531 definitely_dead 0 2024-03-30T13:14:56.462 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.4. 2024-03-30T13:14:56.487 INFO:tasks.thrashosds.thrasher:dead_osds=[4], live_osds=[0, 2, 6, 3, 7, 1, 5] 2024-03-30T13:14:56.487 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:14:56.488 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:14:56.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:56.606+0000 7f0879d90640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:56.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:14:56.622+0000 7f0879d90640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:14:56.794 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:14:56.794 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:14:56.813 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1107,"stamp":"2024-03-30T13:14:56.457869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":367379263,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":373712,"kb_used_data":127136,"kb_used_omap":43,"kb_used_meta":246420,"kb_avail":660229168,"statfs":{"total":676457349120,"available":676074668032,"internally_reserved":0,"allocated":130187264,"data_stored":105301227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44964,"internal_metadata":252334172},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.050301"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1223,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-03-30T13:14:53.758809+0000","last_change":"2024-03-30T13:14:38.236713+0000","last_active":"2024-03-30T13:14:53.758809+0000","last_peered":"2024-03-30T13:14:53.758809+0000","last_clean":"2024-03-30T13:14:53.758809+0000","last_became_active":"2024-03-30T13:14:38.236046+0000","last_became_peered":"2024-03-30T13:14:38.236046+0000","last_unstale":"2024-03-30T13:14:53.758809+0000","last_undegraded":"2024-03-30T13:14:53.758809+0000","last_fullsized":"2024-03-30T13:14:53.758809+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:14:07.953391+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:14:07.953391+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:22.322503+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":22941225,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1145,"reported_epoch":531,"state":"stale+active+clean","last_fresh":"2024-03-30T13:14:53.430992+0000","last_change":"2024-03-30T13:14:50.629414+0000","last_active":"2024-03-30T13:14:53.430992+0000","last_peered":"2024-03-30T13:14:53.430992+0000","last_clean":"2024-03-30T13:14:53.430992+0000","last_became_active":"2024-03-30T13:12:00.216256+0000","last_became_peered":"2024-03-30T13:12:00.216256+0000","last_unstale":"2024-03-30T13:14:56.457890+0000","last_undegraded":"2024-03-30T13:14:53.430992+0000","last_fullsized":"2024-03-30T13:14:53.430992+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:14:50.629162+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:09.043984+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":745,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-03-30T13:14:52.419873+0000","last_change":"2024-03-30T13:13:28.025038+0000","last_active":"2024-03-30T13:14:52.419873+0000","last_peered":"2024-03-30T13:14:52.419873+0000","last_clean":"2024-03-30T13:14:52.419873+0000","last_became_active":"2024-03-30T13:12:51.212130+0000","last_became_peered":"2024-03-30T13:12:51.212130+0000","last_unstale":"2024-03-30T13:14:52.419873+0000","last_undegraded":"2024-03-30T13:14:52.419873+0000","last_fullsized":"2024-03-30T13:14:52.419873+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:13:28.024596+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:13:28.024596+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":687,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-03-30T13:14:52.419473+0000","last_change":"2024-03-30T13:13:47.119549+0000","last_active":"2024-03-30T13:14:52.419473+0000","last_peered":"2024-03-30T13:14:52.419473+0000","last_clean":"2024-03-30T13:14:52.419473+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:14:52.419473+0000","last_undegraded":"2024-03-30T13:14:52.419473+0000","last_fullsized":"2024-03-30T13:14:52.419473+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:13:47.119125+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:13:47.119125+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:15.963858+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":61,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-03-30T13:14:51.557042+0000","last_change":"2024-03-30T13:14:38.234542+0000","last_active":"2024-03-30T13:14:51.557042+0000","last_peered":"2024-03-30T13:14:51.557042+0000","last_clean":"2024-03-30T13:14:51.557042+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:14:51.557042+0000","last_undegraded":"2024-03-30T13:14:51.557042+0000","last_fullsized":"2024-03-30T13:14:51.557042+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:49.332375+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:13:49.332375+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:19.045572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1225,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-03-30T13:14:51.554418+0000","last_change":"2024-03-30T13:14:38.238826+0000","last_active":"2024-03-30T13:14:51.554418+0000","last_peered":"2024-03-30T13:14:51.554418+0000","last_clean":"2024-03-30T13:14:51.554418+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:14:51.554418+0000","last_undegraded":"2024-03-30T13:14:51.554418+0000","last_fullsized":"2024-03-30T13:14:51.554418+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:13:35.802156+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:13:35.802156+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:14:54.127627+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":730,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-03-30T13:14:52.419544+0000","last_change":"2024-03-30T13:14:26.237658+0000","last_active":"2024-03-30T13:14:52.419544+0000","last_peered":"2024-03-30T13:14:52.419544+0000","last_clean":"2024-03-30T13:14:52.419544+0000","last_became_active":"2024-03-30T13:12:00.202949+0000","last_became_peered":"2024-03-30T13:12:00.202949+0000","last_unstale":"2024-03-30T13:14:52.419544+0000","last_undegraded":"2024-03-30T13:14:52.419544+0000","last_fullsized":"2024-03-30T13:14:52.419544+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:14:26.237251+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:14:26.237251+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:30.208212+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":497,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-03-30T13:14:53.429937+0000","last_change":"2024-03-30T13:14:39.618860+0000","last_active":"2024-03-30T13:14:53.429937+0000","last_peered":"2024-03-30T13:14:53.429937+0000","last_clean":"2024-03-30T13:14:53.429937+0000","last_became_active":"2024-03-30T13:14:38.233325+0000","last_became_peered":"2024-03-30T13:14:38.233325+0000","last_unstale":"2024-03-30T13:14:53.429937+0000","last_undegraded":"2024-03-30T13:14:53.429937+0000","last_fullsized":"2024-03-30T13:14:53.429937+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:14:39.618688+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:14:39.618688+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:00.662395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1333,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-03-30T13:14:51.554336+0000","last_change":"2024-03-30T13:14:38.238691+0000","last_active":"2024-03-30T13:14:51.554336+0000","last_peered":"2024-03-30T13:14:51.554336+0000","last_clean":"2024-03-30T13:14:51.554336+0000","last_became_active":"2024-03-30T13:14:38.238174+0000","last_became_peered":"2024-03-30T13:14:38.238174+0000","last_unstale":"2024-03-30T13:14:51.554336+0000","last_undegraded":"2024-03-30T13:14:51.554336+0000","last_fullsized":"2024-03-30T13:14:51.554336+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:14:24.557218+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:14:24.557218+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:30.601618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":71059072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1019,"reported_epoch":531,"state":"stale+active+clean","last_fresh":"2024-03-30T13:14:53.430747+0000","last_change":"2024-03-30T13:14:49.668300+0000","last_active":"2024-03-30T13:14:53.430747+0000","last_peered":"2024-03-30T13:14:53.430747+0000","last_clean":"2024-03-30T13:14:53.430747+0000","last_became_active":"2024-03-30T13:12:00.203605+0000","last_became_peered":"2024-03-30T13:12:00.203605+0000","last_unstale":"2024-03-30T13:14:56.457893+0000","last_undegraded":"2024-03-30T13:14:53.430747+0000","last_fullsized":"2024-03-30T13:14:53.430747+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:14:49.668025+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:49.668025+0000","last_clean_scrub_stamp":"2024-03-30T13:14:49.668025+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:52.350953+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1195,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-03-30T13:14:53.758548+0000","last_change":"2024-03-30T13:13:57.067928+0000","last_active":"2024-03-30T13:14:53.758548+0000","last_peered":"2024-03-30T13:14:53.758548+0000","last_clean":"2024-03-30T13:14:53.758548+0000","last_became_active":"2024-03-30T13:12:00.202952+0000","last_became_peered":"2024-03-30T13:12:00.202952+0000","last_unstale":"2024-03-30T13:14:53.758548+0000","last_undegraded":"2024-03-30T13:14:53.758548+0000","last_fullsized":"2024-03-30T13:14:53.758548+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:13:57.067688+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:13:57.067688+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:13.208052+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1313,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-03-30T13:14:53.758508+0000","last_change":"2024-03-30T13:14:38.238033+0000","last_active":"2024-03-30T13:14:53.758508+0000","last_peered":"2024-03-30T13:14:53.758508+0000","last_clean":"2024-03-30T13:14:53.758508+0000","last_became_active":"2024-03-30T13:14:38.236424+0000","last_became_peered":"2024-03-30T13:14:38.236424+0000","last_unstale":"2024-03-30T13:14:53.758508+0000","last_undegraded":"2024-03-30T13:14:53.758508+0000","last_fullsized":"2024-03-30T13:14:53.758508+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:13:48.913862+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:09:47.821071+0000","last_clean_scrub_stamp":"2024-03-30T13:13:48.913862+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:10.572041+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":76975004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":74,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-03-30T13:14:51.557018+0000","last_change":"2024-03-30T13:14:38.236713+0000","last_active":"2024-03-30T13:14:51.557018+0000","last_peered":"2024-03-30T13:14:51.557018+0000","last_clean":"2024-03-30T13:14:51.557018+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:14:51.557018+0000","last_undegraded":"2024-03-30T13:14:51.557018+0000","last_fullsized":"2024-03-30T13:14:51.557018+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:13:52.331764+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:13:52.331764+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:10.326899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":666,"reported_epoch":530,"state":"active+clean","last_fresh":"2024-03-30T13:14:52.419473+0000","last_change":"2024-03-30T13:14:47.424410+0000","last_active":"2024-03-30T13:14:52.419473+0000","last_peered":"2024-03-30T13:14:52.419473+0000","last_clean":"2024-03-30T13:14:52.419473+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:14:52.419473+0000","last_undegraded":"2024-03-30T13:14:52.419473+0000","last_fullsized":"2024-03-30T13:14:52.419473+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:14:47.424173+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:14:47.424173+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:47.850501+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":58,"reported_epoch":529,"state":"active+clean","last_fresh":"2024-03-30T13:14:51.557085+0000","last_change":"2024-03-30T13:14:38.236720+0000","last_active":"2024-03-30T13:14:51.557085+0000","last_peered":"2024-03-30T13:14:51.557085+0000","last_clean":"2024-03-30T13:14:51.557085+0000","last_became_active":"2024-03-30T13:14:38.235512+0000","last_became_peered":"2024-03-30T13:14:38.235512+0000","last_unstale":"2024-03-30T13:14:51.557085+0000","last_undegraded":"2024-03-30T13:14:51.557085+0000","last_fullsized":"2024-03-30T13:14:51.557085+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:14:13.517128+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:14:13.517128+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:36.511883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":439,"reported_epoch":531,"state":"active+clean","last_fresh":"2024-03-30T13:14:53.429819+0000","last_change":"2024-03-30T13:14:38.231723+0000","last_active":"2024-03-30T13:14:53.429819+0000","last_peered":"2024-03-30T13:14:53.429819+0000","last_clean":"2024-03-30T13:14:53.429819+0000","last_became_active":"2024-03-30T13:14:38.231065+0000","last_became_peered":"2024-03-30T13:14:38.231065+0000","last_unstale":"2024-03-30T13:14:53.429819+0000","last_undegraded":"2024-03-30T13:14:53.429819+0000","last_fullsized":"2024-03-30T13:14:53.429819+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:14:07.270866+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:14:07.270866+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:09.462813+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15680733,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":150,"num_bytes_recovered":367379263,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834180,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57736,"kb_used_data":21432,"kb_used_omap":5,"kb_used_meta":36282,"kb_avail":94314104,"statfs":{"total":96636764160,"available":96577642496,"internally_reserved":0,"allocated":21946368,"data_stored":18386362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":37152866},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157445,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63796,"kb_used_data":26152,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94308044,"statfs":{"total":96636764160,"available":96571437056,"internally_reserved":0,"allocated":26779648,"data_stored":23233244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509676,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47992,"kb_used_data":12648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323848,"statfs":{"total":96636764160,"available":96587620352,"internally_reserved":0,"allocated":12951552,"data_stored":9391546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":461,"seq":1979979923482,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48676,"kb_used_data":11924,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94323164,"statfs":{"total":96636764160,"available":96586919936,"internally_reserved":0,"allocated":12210176,"data_stored":8650170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7692,"internal_metadata":37609972},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509677,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51012,"kb_used_data":19824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320828,"statfs":{"total":96636764160,"available":96584527872,"internally_reserved":0,"allocated":20299776,"data_stored":16739770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297071,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54000,"kb_used_data":16356,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94317840,"statfs":{"total":96636764160,"available":96581468160,"internally_reserved":0,"allocated":16748544,"data_stored":13195339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147385,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50500,"kb_used_data":18800,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321340,"statfs":{"total":96636764160,"available":96585052160,"internally_reserved":0,"allocated":19251200,"data_stored":15704796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:14:56.814 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:15:21.815 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:15:21.815 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:15:21.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:21.942+0000 7f9599dd4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:21.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:21.954+0000 7f9599dd4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:22.131 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:15:22.131 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:15:22.150 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1121,"stamp":"2024-03-30T13:15:22.126743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":407160904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":386096,"kb_used_data":139576,"kb_used_omap":43,"kb_used_meta":246420,"kb_avail":660216784,"statfs":{"total":676457349120,"available":676061986816,"internally_reserved":0,"allocated":142925824,"data_stored":117912024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44964,"internal_metadata":252334172},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004128"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1239,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-30T13:14:58.325041+0000","last_change":"2024-03-30T13:14:58.325041+0000","last_active":"2024-03-30T13:14:58.325041+0000","last_peered":"2024-03-30T13:14:58.325041+0000","last_clean":"2024-03-30T13:14:58.325041+0000","last_became_active":"2024-03-30T13:14:57.476487+0000","last_became_peered":"2024-03-30T13:14:57.476487+0000","last_unstale":"2024-03-30T13:14:58.325041+0000","last_undegraded":"2024-03-30T13:14:58.325041+0000","last_fullsized":"2024-03-30T13:14:58.325041+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:14:07.953391+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:14:07.953391+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:32.125168+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":27529470,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":15,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-30T13:14:57.514819+0000","last_change":"2024-03-30T13:14:57.513171+0000","last_active":"2024-03-30T13:14:57.514819+0000","last_peered":"2024-03-30T13:14:57.514819+0000","last_clean":"2024-03-30T13:14:57.514819+0000","last_became_active":"2024-03-30T13:14:57.483391+0000","last_became_peered":"2024-03-30T13:14:57.483391+0000","last_unstale":"2024-03-30T13:14:57.514819+0000","last_undegraded":"2024-03-30T13:14:57.514819+0000","last_fullsized":"2024-03-30T13:14:57.514819+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:14:50.629162+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:20.398685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":774,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-30T13:15:03.464646+0000","last_change":"2024-03-30T13:15:03.464718+0000","last_active":"2024-03-30T13:15:03.464646+0000","last_peered":"2024-03-30T13:15:03.464646+0000","last_clean":"2024-03-30T13:15:03.464646+0000","last_became_active":"2024-03-30T13:14:57.482626+0000","last_became_peered":"2024-03-30T13:14:57.482626+0000","last_unstale":"2024-03-30T13:15:03.464646+0000","last_undegraded":"2024-03-30T13:15:03.464646+0000","last_fullsized":"2024-03-30T13:15:03.464646+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:15:03.464473+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:15:03.464473+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:18.720618+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":39075780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":702,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-30T13:15:16.348776+0000","last_change":"2024-03-30T13:15:16.348851+0000","last_active":"2024-03-30T13:15:16.348776+0000","last_peered":"2024-03-30T13:15:16.348776+0000","last_clean":"2024-03-30T13:15:16.348776+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:15:16.348776+0000","last_undegraded":"2024-03-30T13:15:16.348776+0000","last_fullsized":"2024-03-30T13:15:16.348776+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:16.348578+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:15:16.348578+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:16.882619+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":68,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-30T13:14:57.468069+0000","last_change":"2024-03-30T13:14:38.234542+0000","last_active":"2024-03-30T13:14:57.468069+0000","last_peered":"2024-03-30T13:14:57.468069+0000","last_clean":"2024-03-30T13:14:57.468069+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:14:57.468069+0000","last_undegraded":"2024-03-30T13:14:57.468069+0000","last_fullsized":"2024-03-30T13:14:57.468069+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:13:49.332375+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:13:49.332375+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:19.045572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1241,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-30T13:14:58.602104+0000","last_change":"2024-03-30T13:14:58.602202+0000","last_active":"2024-03-30T13:14:58.602104+0000","last_peered":"2024-03-30T13:14:58.602104+0000","last_clean":"2024-03-30T13:14:58.602104+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:14:58.602104+0000","last_undegraded":"2024-03-30T13:14:58.602104+0000","last_fullsized":"2024-03-30T13:14:58.602104+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:14:58.601881+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:14:58.601881+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:18.959306+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":757,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-30T13:14:57.626754+0000","last_change":"2024-03-30T13:14:57.625840+0000","last_active":"2024-03-30T13:14:57.626754+0000","last_peered":"2024-03-30T13:14:57.626754+0000","last_clean":"2024-03-30T13:14:57.626754+0000","last_became_active":"2024-03-30T13:14:57.482457+0000","last_became_peered":"2024-03-30T13:14:57.482457+0000","last_unstale":"2024-03-30T13:14:57.626754+0000","last_undegraded":"2024-03-30T13:14:57.626754+0000","last_fullsized":"2024-03-30T13:14:57.626754+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:14:26.237251+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:14:26.237251+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:47.031968+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":43916097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":501,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-30T13:14:57.466357+0000","last_change":"2024-03-30T13:14:39.618860+0000","last_active":"2024-03-30T13:14:57.466357+0000","last_peered":"2024-03-30T13:14:57.466357+0000","last_clean":"2024-03-30T13:14:57.466357+0000","last_became_active":"2024-03-30T13:14:38.233325+0000","last_became_peered":"2024-03-30T13:14:38.233325+0000","last_unstale":"2024-03-30T13:14:57.466357+0000","last_undegraded":"2024-03-30T13:14:57.466357+0000","last_fullsized":"2024-03-30T13:14:57.466357+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:14:39.618688+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:14:39.618688+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:00.662395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1341,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-30T13:14:57.478308+0000","last_change":"2024-03-30T13:14:38.238691+0000","last_active":"2024-03-30T13:14:57.478308+0000","last_peered":"2024-03-30T13:14:57.478308+0000","last_clean":"2024-03-30T13:14:57.478308+0000","last_became_active":"2024-03-30T13:14:38.238174+0000","last_became_peered":"2024-03-30T13:14:38.238174+0000","last_unstale":"2024-03-30T13:14:57.478308+0000","last_undegraded":"2024-03-30T13:14:57.478308+0000","last_fullsized":"2024-03-30T13:14:57.478308+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:14:24.557218+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:14:24.557218+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:30.601618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":71059072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":15,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-30T13:14:57.514313+0000","last_change":"2024-03-30T13:14:57.513112+0000","last_active":"2024-03-30T13:14:57.514313+0000","last_peered":"2024-03-30T13:14:57.514313+0000","last_clean":"2024-03-30T13:14:57.514313+0000","last_became_active":"2024-03-30T13:14:57.481083+0000","last_became_peered":"2024-03-30T13:14:57.481083+0000","last_unstale":"2024-03-30T13:14:57.514313+0000","last_undegraded":"2024-03-30T13:14:57.514313+0000","last_fullsized":"2024-03-30T13:14:57.514313+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:14:49.668025+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:49.668025+0000","last_clean_scrub_stamp":"2024-03-30T13:14:49.668025+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:58.053559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1217,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-30T13:14:57.594479+0000","last_change":"2024-03-30T13:14:57.594479+0000","last_active":"2024-03-30T13:14:57.594479+0000","last_peered":"2024-03-30T13:14:57.594479+0000","last_clean":"2024-03-30T13:14:57.594479+0000","last_became_active":"2024-03-30T13:14:57.487678+0000","last_became_peered":"2024-03-30T13:14:57.487678+0000","last_unstale":"2024-03-30T13:14:57.594479+0000","last_undegraded":"2024-03-30T13:14:57.594479+0000","last_fullsized":"2024-03-30T13:14:57.594479+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:13:57.067688+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:13:57.067688+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:26.911421+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25479082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1325,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-30T13:15:12.093133+0000","last_change":"2024-03-30T13:15:12.093216+0000","last_active":"2024-03-30T13:15:12.093133+0000","last_peered":"2024-03-30T13:15:12.093133+0000","last_clean":"2024-03-30T13:15:12.093133+0000","last_became_active":"2024-03-30T13:14:38.236424+0000","last_became_peered":"2024-03-30T13:14:38.236424+0000","last_unstale":"2024-03-30T13:15:12.093133+0000","last_undegraded":"2024-03-30T13:15:12.093133+0000","last_fullsized":"2024-03-30T13:15:12.093133+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:15:12.092921+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:34.616328+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":76975004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":89,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-30T13:15:14.319993+0000","last_change":"2024-03-30T13:15:14.320075+0000","last_active":"2024-03-30T13:15:14.319993+0000","last_peered":"2024-03-30T13:15:14.319993+0000","last_clean":"2024-03-30T13:15:14.319993+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:15:14.319993+0000","last_undegraded":"2024-03-30T13:15:14.319993+0000","last_fullsized":"2024-03-30T13:15:14.319993+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:15:14.319813+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:15:14.319813+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:34.069151+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":672,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-30T13:14:57.472021+0000","last_change":"2024-03-30T13:14:47.424410+0000","last_active":"2024-03-30T13:14:57.472021+0000","last_peered":"2024-03-30T13:14:57.472021+0000","last_clean":"2024-03-30T13:14:57.472021+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:14:57.472021+0000","last_undegraded":"2024-03-30T13:14:57.472021+0000","last_fullsized":"2024-03-30T13:14:57.472021+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:14:47.424173+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:14:47.424173+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:47.850501+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":65,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-30T13:14:57.468093+0000","last_change":"2024-03-30T13:14:38.236720+0000","last_active":"2024-03-30T13:14:57.468093+0000","last_peered":"2024-03-30T13:14:57.468093+0000","last_clean":"2024-03-30T13:14:57.468093+0000","last_became_active":"2024-03-30T13:14:38.235512+0000","last_became_peered":"2024-03-30T13:14:38.235512+0000","last_unstale":"2024-03-30T13:14:57.468093+0000","last_undegraded":"2024-03-30T13:14:57.468093+0000","last_fullsized":"2024-03-30T13:14:57.468093+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:14:13.517128+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:14:13.517128+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:36.511883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":452,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-03-30T13:15:10.510498+0000","last_change":"2024-03-30T13:15:10.510582+0000","last_active":"2024-03-30T13:15:10.510498+0000","last_peered":"2024-03-30T13:15:10.510498+0000","last_clean":"2024-03-30T13:15:10.510498+0000","last_became_active":"2024-03-30T13:14:38.231065+0000","last_became_peered":"2024-03-30T13:14:38.231065+0000","last_unstale":"2024-03-30T13:15:10.510498+0000","last_undegraded":"2024-03-30T13:15:10.510498+0000","last_fullsized":"2024-03-30T13:15:10.510498+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:10.510312+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:15:10.510312+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:36.027524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15680733,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":407160904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":109256704,"data_stored":109256704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834185,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57752,"kb_used_data":21456,"kb_used_omap":5,"kb_used_meta":36282,"kb_avail":94314088,"statfs":{"total":96636764160,"available":96577626112,"internally_reserved":0,"allocated":21970944,"data_stored":18397544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":37152866},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157450,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63840,"kb_used_data":26200,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94308000,"statfs":{"total":96636764160,"available":96571392000,"internally_reserved":0,"allocated":26828800,"data_stored":23255400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509681,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54764,"kb_used_data":19428,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94317076,"statfs":{"total":96636764160,"available":96580685824,"internally_reserved":0,"allocated":19894272,"data_stored":16320872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":461,"seq":1979979923487,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49784,"kb_used_data":13040,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94322056,"statfs":{"total":96636764160,"available":96585785344,"internally_reserved":0,"allocated":13352960,"data_stored":9779560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7692,"internal_metadata":37609972},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509682,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51528,"kb_used_data":20352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320312,"statfs":{"total":96636764160,"available":96583999488,"internally_reserved":0,"allocated":20840448,"data_stored":17267048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297076,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54032,"kb_used_data":16392,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94317808,"statfs":{"total":96636764160,"available":96581435392,"internally_reserved":0,"allocated":16785408,"data_stored":13212008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147390,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54396,"kb_used_data":22708,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94317444,"statfs":{"total":96636764160,"available":96581062656,"internally_reserved":0,"allocated":23252992,"data_stored":19679592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14340096,"data_stored":14340096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6852608,"data_stored":6852608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13393920,"data_stored":13393920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16752640,"data_stored":16752640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:15:22.151 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:15:22.152 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:15:22.152 INFO:tasks.thrashosds.thrasher:Reviving osd 4 2024-03-30T13:15:22.152 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-30T13:15:22.152 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-30T13:15:22.155 INFO:tasks.ceph.osd.4:Started 2024-03-30T13:15:22.155 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-30T13:15:22.225 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:15:22.225+0000 7f94841a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:22.234 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:15:22.233+0000 7f94841a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:22.234 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:15:22.233+0000 7f94841a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:22.272 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T13:15:22.272 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T13:15:22.272 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T13:15:22.272 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T13:15:22.272 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T13:15:22.272 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T13:15:22.272 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-03-30T13:15:22.273 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-03-30T13:15:22.273 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-03-30T13:15:22.273 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-03-30T13:15:22.273 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-03-30T13:15:22.273 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T13:15:22.274 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:15:22.274 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-03-30T13:15:22.793 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:15:22.793+0000 7f94841a17c0 -1 Falling back to public interface 2024-03-30T13:15:26.578 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:15:26.577+0000 7f94841a17c0 -1 osd.4 531 log_to_monitors true 2024-03-30T13:15:27.275 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-30T13:15:27.283 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:15:27.285+0000 7f947bf76640 -1 osd.4 531 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:15:27.406 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T13:15:27.406 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T13:15:27.406 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T13:15:27.406 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T13:15:27.419 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:15:27.419 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-03-30T13:15:27.419 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 4 2024-03-30T13:15:27.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:27.545+0000 7f8adbf7f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:27.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:27.561+0000 7f8adbf7f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:27.744 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:15:27.741+0000 7fbb38f08640 -1 mon.a@0(leader).osd e535 definitely_dead 0 2024-03-30T13:15:28.237 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.4. 2024-03-30T13:15:28.274 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-03-30T13:15:28.275 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:15:28.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:28.389+0000 7f3a42b90640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:28.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:28.409+0000 7f3a42b90640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:29.272 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:15:29.298 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:15:29.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:29.413+0000 7f89e215e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:29.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:29.429+0000 7f89e215e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:30.278 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:15:30.303 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:15:30.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:30.421+0000 7f232387f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:30.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:30.437+0000 7f232387f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:31.280 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:15:31.305 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:15:31.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:31.421+0000 7f6a9ef78640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:31.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:31.437+0000 7f6a9ef78640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:32.296 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:15:32.318 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:15:32.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:32.433+0000 7f249949b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:32.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:32.449+0000 7f249949b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:33.308 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:15:33.332 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:15:33.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:33.449+0000 7f0574c74640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:33.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:33.465+0000 7f0574c74640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:34.312 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:15:34.337 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:15:34.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:34.457+0000 7f95680c6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:34.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:34.473+0000 7f95680c6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:35.334 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:15:35.359 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:15:35.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:35.477+0000 7f88d7293640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:35.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:35.493+0000 7f88d7293640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:36.343 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:15:36.366 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:15:36.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:36.485+0000 7efd8139a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:36.501+0000 7efd8139a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:37.361 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:15:37.388 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:15:37.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:37.509+0000 7f4ce6b65640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:37.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:37.525+0000 7f4ce6b65640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:38.364 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:15:38.390 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:15:38.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:38.509+0000 7f3d684df640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:38.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:38.525+0000 7f3d684df640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:39.369 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:15:39.398 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:15:39.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:39.517+0000 7f51e939b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:39.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:39.537+0000 7f51e939b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:40.387 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:15:40.413 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:15:40.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:40.533+0000 7fd9c642e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:40.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:40.549+0000 7fd9c642e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:41.399 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:15:41.426 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:15:41.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:41.545+0000 7f8e585ed640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:41.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:41.565+0000 7f8e585ed640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:42.414 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:15:42.439 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:15:42.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:42.557+0000 7f6a68c35640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:42.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:42.573+0000 7f6a68c35640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:43.425 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:15:43.450 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:15:43.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:43.565+0000 7f906b53b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:43.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:43.585+0000 7f906b53b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:44.436 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:15:44.462 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:15:44.463 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:15:44.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:44.585+0000 7f6e70397640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:44.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:44.601+0000 7f6e70397640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:44.774 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:15:44.775 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:15:44.793 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1151,"stamp":"2024-03-30T13:15:44.441694+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":407160904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":430876,"kb_used_data":146648,"kb_used_omap":54,"kb_used_meta":284041,"kb_avail":754543844,"statfs":{"total":773094113280,"available":772652896256,"internally_reserved":0,"allocated":150167552,"data_stored":120812195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55955,"internal_metadata":290858349},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.049435"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1283,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-03-30T13:15:41.196737+0000","last_change":"2024-03-30T13:15:31.536644+0000","last_active":"2024-03-30T13:15:41.196737+0000","last_peered":"2024-03-30T13:15:41.196737+0000","last_clean":"2024-03-30T13:15:41.196737+0000","last_became_active":"2024-03-30T13:15:29.285871+0000","last_became_peered":"2024-03-30T13:15:29.285871+0000","last_unstale":"2024-03-30T13:15:41.196737+0000","last_undegraded":"2024-03-30T13:15:41.196737+0000","last_fullsized":"2024-03-30T13:15:41.196737+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:31.536364+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:15:31.536364+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:45.858456+0000","scrub_duration":515,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":27529470,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1177,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-30T13:15:42.413284+0000","last_change":"2024-03-30T13:15:29.524842+0000","last_active":"2024-03-30T13:15:42.413284+0000","last_peered":"2024-03-30T13:15:42.413284+0000","last_clean":"2024-03-30T13:15:42.413284+0000","last_became_active":"2024-03-30T13:15:29.523936+0000","last_became_peered":"2024-03-30T13:15:29.523936+0000","last_unstale":"2024-03-30T13:15:42.413284+0000","last_undegraded":"2024-03-30T13:15:42.413284+0000","last_fullsized":"2024-03-30T13:15:42.413284+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:14:50.629162+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:56.386997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":813,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-30T13:15:42.416457+0000","last_change":"2024-03-30T13:15:29.285894+0000","last_active":"2024-03-30T13:15:42.416457+0000","last_peered":"2024-03-30T13:15:42.416457+0000","last_clean":"2024-03-30T13:15:42.416457+0000","last_became_active":"2024-03-30T13:15:29.284839+0000","last_became_peered":"2024-03-30T13:15:29.284839+0000","last_unstale":"2024-03-30T13:15:42.416457+0000","last_undegraded":"2024-03-30T13:15:42.416457+0000","last_fullsized":"2024-03-30T13:15:42.416457+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:15:03.464473+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:15:03.464473+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:31.803707+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":39075780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":734,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-30T13:15:42.416394+0000","last_change":"2024-03-30T13:15:16.348851+0000","last_active":"2024-03-30T13:15:42.416394+0000","last_peered":"2024-03-30T13:15:42.416394+0000","last_clean":"2024-03-30T13:15:42.416394+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:15:42.416394+0000","last_undegraded":"2024-03-30T13:15:42.416394+0000","last_fullsized":"2024-03-30T13:15:42.416394+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:16.348578+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:15:16.348578+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:16.882619+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":107,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-30T13:15:41.700579+0000","last_change":"2024-03-30T13:15:21.424552+0000","last_active":"2024-03-30T13:15:41.700579+0000","last_peered":"2024-03-30T13:15:41.700579+0000","last_clean":"2024-03-30T13:15:41.700579+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:15:41.700579+0000","last_undegraded":"2024-03-30T13:15:41.700579+0000","last_fullsized":"2024-03-30T13:15:41.700579+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:15:21.424313+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:15:21.424313+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:49.667982+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1270,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-30T13:15:41.627090+0000","last_change":"2024-03-30T13:14:58.602202+0000","last_active":"2024-03-30T13:15:41.627090+0000","last_peered":"2024-03-30T13:15:41.627090+0000","last_clean":"2024-03-30T13:15:41.627090+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:15:41.627090+0000","last_undegraded":"2024-03-30T13:15:41.627090+0000","last_fullsized":"2024-03-30T13:15:41.627090+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:14:58.601881+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:14:58.601881+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:18.959306+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":803,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-30T13:15:42.416408+0000","last_change":"2024-03-30T13:15:31.305638+0000","last_active":"2024-03-30T13:15:42.416408+0000","last_peered":"2024-03-30T13:15:42.416408+0000","last_clean":"2024-03-30T13:15:42.416408+0000","last_became_active":"2024-03-30T13:15:29.283073+0000","last_became_peered":"2024-03-30T13:15:29.283073+0000","last_unstale":"2024-03-30T13:15:42.416408+0000","last_undegraded":"2024-03-30T13:15:42.416408+0000","last_fullsized":"2024-03-30T13:15:42.416408+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:15:31.305358+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:15:31.305358+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:47.066127+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":43916097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":529,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-03-30T13:15:39.373828+0000","last_change":"2024-03-30T13:14:39.618860+0000","last_active":"2024-03-30T13:15:39.373828+0000","last_peered":"2024-03-30T13:15:39.373828+0000","last_clean":"2024-03-30T13:15:39.373828+0000","last_became_active":"2024-03-30T13:14:38.233325+0000","last_became_peered":"2024-03-30T13:14:38.233325+0000","last_unstale":"2024-03-30T13:15:39.373828+0000","last_undegraded":"2024-03-30T13:15:39.373828+0000","last_fullsized":"2024-03-30T13:15:39.373828+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:14:39.618688+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:14:39.618688+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:00.662395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1378,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-30T13:15:41.627150+0000","last_change":"2024-03-30T13:15:36.454725+0000","last_active":"2024-03-30T13:15:41.627150+0000","last_peered":"2024-03-30T13:15:41.627150+0000","last_clean":"2024-03-30T13:15:41.627150+0000","last_became_active":"2024-03-30T13:14:38.238174+0000","last_became_peered":"2024-03-30T13:14:38.238174+0000","last_unstale":"2024-03-30T13:15:41.627150+0000","last_undegraded":"2024-03-30T13:15:41.627150+0000","last_fullsized":"2024-03-30T13:15:41.627150+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:36.454575+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:15:36.454575+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:40.210170+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":71059072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1049,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-30T13:15:42.413147+0000","last_change":"2024-03-30T13:15:29.284525+0000","last_active":"2024-03-30T13:15:42.413147+0000","last_peered":"2024-03-30T13:15:42.413147+0000","last_clean":"2024-03-30T13:15:42.413147+0000","last_became_active":"2024-03-30T13:15:29.282697+0000","last_became_peered":"2024-03-30T13:15:29.282697+0000","last_unstale":"2024-03-30T13:15:42.413147+0000","last_undegraded":"2024-03-30T13:15:42.413147+0000","last_fullsized":"2024-03-30T13:15:42.413147+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:14:49.668025+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:49.668025+0000","last_clean_scrub_stamp":"2024-03-30T13:14:49.668025+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:59.022759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1260,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-03-30T13:15:41.196582+0000","last_change":"2024-03-30T13:15:29.284976+0000","last_active":"2024-03-30T13:15:41.196582+0000","last_peered":"2024-03-30T13:15:41.196582+0000","last_clean":"2024-03-30T13:15:41.196582+0000","last_became_active":"2024-03-30T13:15:29.283557+0000","last_became_peered":"2024-03-30T13:15:29.283557+0000","last_unstale":"2024-03-30T13:15:41.196582+0000","last_undegraded":"2024-03-30T13:15:41.196582+0000","last_fullsized":"2024-03-30T13:15:41.196582+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:15:27.030418+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:15:27.030418+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:52.751419+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25479082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1354,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-03-30T13:15:41.196557+0000","last_change":"2024-03-30T13:15:12.093216+0000","last_active":"2024-03-30T13:15:41.196557+0000","last_peered":"2024-03-30T13:15:41.196557+0000","last_clean":"2024-03-30T13:15:41.196557+0000","last_became_active":"2024-03-30T13:14:38.236424+0000","last_became_peered":"2024-03-30T13:14:38.236424+0000","last_unstale":"2024-03-30T13:15:41.196557+0000","last_undegraded":"2024-03-30T13:15:41.196557+0000","last_fullsized":"2024-03-30T13:15:41.196557+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:15:12.092921+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:34.616328+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":76975004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":120,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-30T13:15:41.700569+0000","last_change":"2024-03-30T13:15:14.320075+0000","last_active":"2024-03-30T13:15:41.700569+0000","last_peered":"2024-03-30T13:15:41.700569+0000","last_clean":"2024-03-30T13:15:41.700569+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:15:41.700569+0000","last_undegraded":"2024-03-30T13:15:41.700569+0000","last_fullsized":"2024-03-30T13:15:41.700569+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:15:14.319813+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:15:14.319813+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:34.069151+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":705,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-30T13:15:42.416375+0000","last_change":"2024-03-30T13:14:47.424410+0000","last_active":"2024-03-30T13:15:42.416375+0000","last_peered":"2024-03-30T13:15:42.416375+0000","last_clean":"2024-03-30T13:15:42.416375+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:15:42.416375+0000","last_undegraded":"2024-03-30T13:15:42.416375+0000","last_fullsized":"2024-03-30T13:15:42.416375+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:14:47.424173+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:14:47.424173+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:47.850501+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":104,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-30T13:15:41.700579+0000","last_change":"2024-03-30T13:15:39.529282+0000","last_active":"2024-03-30T13:15:41.700579+0000","last_peered":"2024-03-30T13:15:41.700579+0000","last_clean":"2024-03-30T13:15:41.700579+0000","last_became_active":"2024-03-30T13:14:38.235512+0000","last_became_peered":"2024-03-30T13:14:38.235512+0000","last_unstale":"2024-03-30T13:15:41.700579+0000","last_undegraded":"2024-03-30T13:15:41.700579+0000","last_fullsized":"2024-03-30T13:15:41.700579+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:15:39.528997+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:15:39.528997+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:00.848645+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":480,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-03-30T13:15:39.373691+0000","last_change":"2024-03-30T13:15:10.510582+0000","last_active":"2024-03-30T13:15:39.373691+0000","last_peered":"2024-03-30T13:15:39.373691+0000","last_clean":"2024-03-30T13:15:39.373691+0000","last_became_active":"2024-03-30T13:14:38.231065+0000","last_became_peered":"2024-03-30T13:14:38.231065+0000","last_unstale":"2024-03-30T13:15:39.373691+0000","last_undegraded":"2024-03-30T13:15:39.373691+0000","last_fullsized":"2024-03-30T13:15:39.373691+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:10.510312+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:15:10.510312+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:36.027524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15680733,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":407160904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834189,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57932,"kb_used_data":21624,"kb_used_omap":5,"kb_used_meta":36282,"kb_avail":94313908,"statfs":{"total":96636764160,"available":96577441792,"internally_reserved":0,"allocated":22142976,"data_stored":18476063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":37152866},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157455,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64028,"kb_used_data":26380,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94307812,"statfs":{"total":96636764160,"available":96571199488,"internally_reserved":0,"allocated":27013120,"data_stored":23339506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":536,"seq":2302102470661,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55752,"kb_used_data":18108,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94316088,"statfs":{"total":96636764160,"available":96579674112,"internally_reserved":0,"allocated":18542592,"data_stored":14855376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10991,"internal_metadata":38524177},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509685,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48188,"kb_used_data":12840,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323652,"statfs":{"total":96636764160,"available":96587419648,"internally_reserved":0,"allocated":13148160,"data_stored":9481247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":461,"seq":1979979923491,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48872,"kb_used_data":12116,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94322968,"statfs":{"total":96636764160,"available":96586719232,"internally_reserved":0,"allocated":12406784,"data_stored":8739871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7692,"internal_metadata":37609972},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509686,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51168,"kb_used_data":19980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320672,"statfs":{"total":96636764160,"available":96584368128,"internally_reserved":0,"allocated":20459520,"data_stored":16812960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297081,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54216,"kb_used_data":16572,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94317624,"statfs":{"total":96636764160,"available":96581246976,"internally_reserved":0,"allocated":16969728,"data_stored":13296114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147395,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50720,"kb_used_data":19028,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321120,"statfs":{"total":96636764160,"available":96584826880,"internally_reserved":0,"allocated":19484672,"data_stored":15811058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:15:44.794 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:15:44.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:44.913+0000 7f38f5be3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:44.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:44.925+0000 7f38f5be3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:45.101 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:15:45.102 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:15:45.120 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1151,"stamp":"2024-03-30T13:15:44.441694+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":407160904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":430876,"kb_used_data":146648,"kb_used_omap":54,"kb_used_meta":284041,"kb_avail":754543844,"statfs":{"total":773094113280,"available":772652896256,"internally_reserved":0,"allocated":150167552,"data_stored":120812195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55955,"internal_metadata":290858349},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.049435"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1283,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-03-30T13:15:41.196737+0000","last_change":"2024-03-30T13:15:31.536644+0000","last_active":"2024-03-30T13:15:41.196737+0000","last_peered":"2024-03-30T13:15:41.196737+0000","last_clean":"2024-03-30T13:15:41.196737+0000","last_became_active":"2024-03-30T13:15:29.285871+0000","last_became_peered":"2024-03-30T13:15:29.285871+0000","last_unstale":"2024-03-30T13:15:41.196737+0000","last_undegraded":"2024-03-30T13:15:41.196737+0000","last_fullsized":"2024-03-30T13:15:41.196737+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:31.536364+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:15:31.536364+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:45.858456+0000","scrub_duration":515,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":27529470,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1177,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-30T13:15:42.413284+0000","last_change":"2024-03-30T13:15:29.524842+0000","last_active":"2024-03-30T13:15:42.413284+0000","last_peered":"2024-03-30T13:15:42.413284+0000","last_clean":"2024-03-30T13:15:42.413284+0000","last_became_active":"2024-03-30T13:15:29.523936+0000","last_became_peered":"2024-03-30T13:15:29.523936+0000","last_unstale":"2024-03-30T13:15:42.413284+0000","last_undegraded":"2024-03-30T13:15:42.413284+0000","last_fullsized":"2024-03-30T13:15:42.413284+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:14:50.629162+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:56.386997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":813,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-30T13:15:42.416457+0000","last_change":"2024-03-30T13:15:29.285894+0000","last_active":"2024-03-30T13:15:42.416457+0000","last_peered":"2024-03-30T13:15:42.416457+0000","last_clean":"2024-03-30T13:15:42.416457+0000","last_became_active":"2024-03-30T13:15:29.284839+0000","last_became_peered":"2024-03-30T13:15:29.284839+0000","last_unstale":"2024-03-30T13:15:42.416457+0000","last_undegraded":"2024-03-30T13:15:42.416457+0000","last_fullsized":"2024-03-30T13:15:42.416457+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:15:03.464473+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:15:03.464473+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:31.803707+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":39075780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":734,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-30T13:15:42.416394+0000","last_change":"2024-03-30T13:15:16.348851+0000","last_active":"2024-03-30T13:15:42.416394+0000","last_peered":"2024-03-30T13:15:42.416394+0000","last_clean":"2024-03-30T13:15:42.416394+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:15:42.416394+0000","last_undegraded":"2024-03-30T13:15:42.416394+0000","last_fullsized":"2024-03-30T13:15:42.416394+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:16.348578+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:15:16.348578+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:16.882619+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":107,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-30T13:15:41.700579+0000","last_change":"2024-03-30T13:15:21.424552+0000","last_active":"2024-03-30T13:15:41.700579+0000","last_peered":"2024-03-30T13:15:41.700579+0000","last_clean":"2024-03-30T13:15:41.700579+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:15:41.700579+0000","last_undegraded":"2024-03-30T13:15:41.700579+0000","last_fullsized":"2024-03-30T13:15:41.700579+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:15:21.424313+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:15:21.424313+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:49.667982+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1270,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-30T13:15:41.627090+0000","last_change":"2024-03-30T13:14:58.602202+0000","last_active":"2024-03-30T13:15:41.627090+0000","last_peered":"2024-03-30T13:15:41.627090+0000","last_clean":"2024-03-30T13:15:41.627090+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:15:41.627090+0000","last_undegraded":"2024-03-30T13:15:41.627090+0000","last_fullsized":"2024-03-30T13:15:41.627090+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:14:58.601881+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:14:58.601881+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:18.959306+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":803,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-30T13:15:42.416408+0000","last_change":"2024-03-30T13:15:31.305638+0000","last_active":"2024-03-30T13:15:42.416408+0000","last_peered":"2024-03-30T13:15:42.416408+0000","last_clean":"2024-03-30T13:15:42.416408+0000","last_became_active":"2024-03-30T13:15:29.283073+0000","last_became_peered":"2024-03-30T13:15:29.283073+0000","last_unstale":"2024-03-30T13:15:42.416408+0000","last_undegraded":"2024-03-30T13:15:42.416408+0000","last_fullsized":"2024-03-30T13:15:42.416408+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:15:31.305358+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:15:31.305358+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:47.066127+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":43916097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":529,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-03-30T13:15:39.373828+0000","last_change":"2024-03-30T13:14:39.618860+0000","last_active":"2024-03-30T13:15:39.373828+0000","last_peered":"2024-03-30T13:15:39.373828+0000","last_clean":"2024-03-30T13:15:39.373828+0000","last_became_active":"2024-03-30T13:14:38.233325+0000","last_became_peered":"2024-03-30T13:14:38.233325+0000","last_unstale":"2024-03-30T13:15:39.373828+0000","last_undegraded":"2024-03-30T13:15:39.373828+0000","last_fullsized":"2024-03-30T13:15:39.373828+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:14:39.618688+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:14:39.618688+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:00.662395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1378,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-30T13:15:41.627150+0000","last_change":"2024-03-30T13:15:36.454725+0000","last_active":"2024-03-30T13:15:41.627150+0000","last_peered":"2024-03-30T13:15:41.627150+0000","last_clean":"2024-03-30T13:15:41.627150+0000","last_became_active":"2024-03-30T13:14:38.238174+0000","last_became_peered":"2024-03-30T13:14:38.238174+0000","last_unstale":"2024-03-30T13:15:41.627150+0000","last_undegraded":"2024-03-30T13:15:41.627150+0000","last_fullsized":"2024-03-30T13:15:41.627150+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:36.454575+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:15:36.454575+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:40.210170+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":71059072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1049,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-30T13:15:42.413147+0000","last_change":"2024-03-30T13:15:29.284525+0000","last_active":"2024-03-30T13:15:42.413147+0000","last_peered":"2024-03-30T13:15:42.413147+0000","last_clean":"2024-03-30T13:15:42.413147+0000","last_became_active":"2024-03-30T13:15:29.282697+0000","last_became_peered":"2024-03-30T13:15:29.282697+0000","last_unstale":"2024-03-30T13:15:42.413147+0000","last_undegraded":"2024-03-30T13:15:42.413147+0000","last_fullsized":"2024-03-30T13:15:42.413147+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:14:49.668025+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:49.668025+0000","last_clean_scrub_stamp":"2024-03-30T13:14:49.668025+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:59.022759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1260,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-03-30T13:15:41.196582+0000","last_change":"2024-03-30T13:15:29.284976+0000","last_active":"2024-03-30T13:15:41.196582+0000","last_peered":"2024-03-30T13:15:41.196582+0000","last_clean":"2024-03-30T13:15:41.196582+0000","last_became_active":"2024-03-30T13:15:29.283557+0000","last_became_peered":"2024-03-30T13:15:29.283557+0000","last_unstale":"2024-03-30T13:15:41.196582+0000","last_undegraded":"2024-03-30T13:15:41.196582+0000","last_fullsized":"2024-03-30T13:15:41.196582+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:15:27.030418+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:15:27.030418+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:52.751419+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25479082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1354,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-03-30T13:15:41.196557+0000","last_change":"2024-03-30T13:15:12.093216+0000","last_active":"2024-03-30T13:15:41.196557+0000","last_peered":"2024-03-30T13:15:41.196557+0000","last_clean":"2024-03-30T13:15:41.196557+0000","last_became_active":"2024-03-30T13:14:38.236424+0000","last_became_peered":"2024-03-30T13:14:38.236424+0000","last_unstale":"2024-03-30T13:15:41.196557+0000","last_undegraded":"2024-03-30T13:15:41.196557+0000","last_fullsized":"2024-03-30T13:15:41.196557+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:15:12.092921+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:34.616328+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":76975004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":120,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-30T13:15:41.700569+0000","last_change":"2024-03-30T13:15:14.320075+0000","last_active":"2024-03-30T13:15:41.700569+0000","last_peered":"2024-03-30T13:15:41.700569+0000","last_clean":"2024-03-30T13:15:41.700569+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:15:41.700569+0000","last_undegraded":"2024-03-30T13:15:41.700569+0000","last_fullsized":"2024-03-30T13:15:41.700569+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:15:14.319813+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:15:14.319813+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:34.069151+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":705,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-30T13:15:42.416375+0000","last_change":"2024-03-30T13:14:47.424410+0000","last_active":"2024-03-30T13:15:42.416375+0000","last_peered":"2024-03-30T13:15:42.416375+0000","last_clean":"2024-03-30T13:15:42.416375+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:15:42.416375+0000","last_undegraded":"2024-03-30T13:15:42.416375+0000","last_fullsized":"2024-03-30T13:15:42.416375+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:14:47.424173+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:14:47.424173+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:47.850501+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":104,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-30T13:15:41.700579+0000","last_change":"2024-03-30T13:15:39.529282+0000","last_active":"2024-03-30T13:15:41.700579+0000","last_peered":"2024-03-30T13:15:41.700579+0000","last_clean":"2024-03-30T13:15:41.700579+0000","last_became_active":"2024-03-30T13:14:38.235512+0000","last_became_peered":"2024-03-30T13:14:38.235512+0000","last_unstale":"2024-03-30T13:15:41.700579+0000","last_undegraded":"2024-03-30T13:15:41.700579+0000","last_fullsized":"2024-03-30T13:15:41.700579+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:15:39.528997+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:15:39.528997+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:00.848645+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":480,"reported_epoch":547,"state":"active+clean","last_fresh":"2024-03-30T13:15:39.373691+0000","last_change":"2024-03-30T13:15:10.510582+0000","last_active":"2024-03-30T13:15:39.373691+0000","last_peered":"2024-03-30T13:15:39.373691+0000","last_clean":"2024-03-30T13:15:39.373691+0000","last_became_active":"2024-03-30T13:14:38.231065+0000","last_became_peered":"2024-03-30T13:14:38.231065+0000","last_unstale":"2024-03-30T13:15:39.373691+0000","last_undegraded":"2024-03-30T13:15:39.373691+0000","last_fullsized":"2024-03-30T13:15:39.373691+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:10.510312+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:15:10.510312+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:36.027524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15680733,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":407160904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834189,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57932,"kb_used_data":21624,"kb_used_omap":5,"kb_used_meta":36282,"kb_avail":94313908,"statfs":{"total":96636764160,"available":96577441792,"internally_reserved":0,"allocated":22142976,"data_stored":18476063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":37152866},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157455,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64028,"kb_used_data":26380,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94307812,"statfs":{"total":96636764160,"available":96571199488,"internally_reserved":0,"allocated":27013120,"data_stored":23339506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":536,"seq":2302102470661,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55752,"kb_used_data":18108,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94316088,"statfs":{"total":96636764160,"available":96579674112,"internally_reserved":0,"allocated":18542592,"data_stored":14855376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10991,"internal_metadata":38524177},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509685,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48188,"kb_used_data":12840,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323652,"statfs":{"total":96636764160,"available":96587419648,"internally_reserved":0,"allocated":13148160,"data_stored":9481247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":461,"seq":1979979923491,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48872,"kb_used_data":12116,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94322968,"statfs":{"total":96636764160,"available":96586719232,"internally_reserved":0,"allocated":12406784,"data_stored":8739871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7692,"internal_metadata":37609972},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509686,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51168,"kb_used_data":19980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320672,"statfs":{"total":96636764160,"available":96584368128,"internally_reserved":0,"allocated":20459520,"data_stored":16812960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297081,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54216,"kb_used_data":16572,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94317624,"statfs":{"total":96636764160,"available":96581246976,"internally_reserved":0,"allocated":16969728,"data_stored":13296114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147395,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50720,"kb_used_data":19028,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321120,"statfs":{"total":96636764160,"available":96584826880,"internally_reserved":0,"allocated":19484672,"data_stored":15811058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:15:45.121 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:15:45.121 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:15:45.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:45.237+0000 7fd0d7f70640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:45.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:45.253+0000 7fd0d7f70640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:45.427 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:15:45.427 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":552,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:15:44.426607+0000","last_up_change":"2024-03-30T13:15:28.233579+0000","last_in_change":"2024-03-30T13:15:28.233579+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":532,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":346,"last_clean_end":457,"up_from":461,"up_thru":515,"down_at":458,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2034332759},{"type":"v1","addr":"172.21.15.67:6818","nonce":2034332759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2034332759},{"type":"v1","addr":"172.21.15.67:6821","nonce":2034332759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2034332759},{"type":"v1","addr":"172.21.15.67:6828","nonce":2034332759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2034332759},{"type":"v1","addr":"172.21.15.67:6825","nonce":2034332759}]},"public_addr":"172.21.15.67:6818/2034332759","cluster_addr":"172.21.15.67:6821/2034332759","heartbeat_back_addr":"172.21.15.67:6828/2034332759","heartbeat_front_addr":"172.21.15.67:6825/2034332759","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":532,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":388,"up_from":393,"up_thru":515,"down_at":389,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6801","nonce":3837194718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6803","nonce":3837194718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6807","nonce":3837194718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6805","nonce":3837194718}]},"public_addr":"172.21.15.67:6801/3837194718","cluster_addr":"172.21.15.67:6803/3837194718","heartbeat_back_addr":"172.21.15.67:6807/3837194718","heartbeat_front_addr":"172.21.15.67:6805/3837194718","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":441,"last_clean_end":531,"up_from":536,"up_thru":536,"down_at":532,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3605867885},{"type":"v1","addr":"172.21.15.82:6803","nonce":3605867885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":3605867885},{"type":"v1","addr":"172.21.15.82:6807","nonce":3605867885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":3605867885},{"type":"v1","addr":"172.21.15.82:6815","nonce":3605867885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":3605867885},{"type":"v1","addr":"172.21.15.82:6811","nonce":3605867885}]},"public_addr":"172.21.15.82:6803/3605867885","cluster_addr":"172.21.15.82:6807/3605867885","heartbeat_back_addr":"172.21.15.82:6815/3605867885","heartbeat_front_addr":"172.21.15.82:6811/3605867885","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":488,"last_clean_end":506,"up_from":515,"up_thru":515,"down_at":507,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6817","nonce":292505453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6819","nonce":292505453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6823","nonce":292505453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6821","nonce":292505453}]},"public_addr":"172.21.15.82:6817/292505453","cluster_addr":"172.21.15.82:6819/292505453","heartbeat_back_addr":"172.21.15.82:6823/292505453","heartbeat_front_addr":"172.21.15.82:6821/292505453","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":227,"last_clean_end":363,"up_from":372,"up_thru":536,"down_at":364,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6801","nonce":189641108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6806","nonce":189641108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6814","nonce":189641108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6810","nonce":189641108}]},"public_addr":"172.21.15.82:6801/189641108","cluster_addr":"172.21.15.82:6806/189641108","heartbeat_back_addr":"172.21.15.82:6814/189641108","heartbeat_front_addr":"172.21.15.82:6810/189641108","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":298,"last_clean_end":409,"up_from":419,"up_thru":536,"down_at":410,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6825","nonce":2166599911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6827","nonce":2166599911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6831","nonce":2166599911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6829","nonce":2166599911}]},"public_addr":"172.21.15.82:6825/2166599911","cluster_addr":"172.21.15.82:6827/2166599911","heartbeat_back_addr":"172.21.15.82:6831/2166599911","heartbeat_front_addr":"172.21.15.82:6829/2166599911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:14:45.583060+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:15:25.237266+0000","dead_epoch":458},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:15:28.906751+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:14:41.722471+0000","dead_epoch":389},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:15:36.571402+0000","dead_epoch":532},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:14:46.777563+0000","dead_epoch":507},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:15:39.514759+0000","dead_epoch":364},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:15:40.057282+0000","dead_epoch":410}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:15:45.445 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:15:45.445 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:15:45.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:45.557+0000 7f1bfd13a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:45.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:45.577+0000 7f1bfd13a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:45.743 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:15:45.761 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:15:45.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:45.861+0000 7f26f4de3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:45.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:45.877+0000 7f26f4de3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:46.049 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:15:46.049 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:15:46.067 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:15:46.067 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:15:46.067 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:15:46.067 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:15:46.067 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 6, 3, 7, 1, 5, 4] 2024-03-30T13:15:46.067 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:15:46.068 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:15:46.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:46.181+0000 7f6cb1d60640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:46.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:46.197+0000 7f6cb1d60640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:46.375 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:15:46.376 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:15:46.394 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1152,"stamp":"2024-03-30T13:15:46.134210+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":407160904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":431040,"kb_used_data":146828,"kb_used_omap":54,"kb_used_meta":284041,"kb_avail":754543680,"statfs":{"total":773094113280,"available":772652728320,"internally_reserved":0,"allocated":150351872,"data_stored":120894850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55955,"internal_metadata":290858349},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.739300"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1283,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-03-30T13:15:41.196737+0000","last_change":"2024-03-30T13:15:31.536644+0000","last_active":"2024-03-30T13:15:41.196737+0000","last_peered":"2024-03-30T13:15:41.196737+0000","last_clean":"2024-03-30T13:15:41.196737+0000","last_became_active":"2024-03-30T13:15:29.285871+0000","last_became_peered":"2024-03-30T13:15:29.285871+0000","last_unstale":"2024-03-30T13:15:41.196737+0000","last_undegraded":"2024-03-30T13:15:41.196737+0000","last_fullsized":"2024-03-30T13:15:41.196737+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:31.536364+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:15:31.536364+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:45.858456+0000","scrub_duration":515,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":27529470,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1177,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-30T13:15:42.413284+0000","last_change":"2024-03-30T13:15:29.524842+0000","last_active":"2024-03-30T13:15:42.413284+0000","last_peered":"2024-03-30T13:15:42.413284+0000","last_clean":"2024-03-30T13:15:42.413284+0000","last_became_active":"2024-03-30T13:15:29.523936+0000","last_became_peered":"2024-03-30T13:15:29.523936+0000","last_unstale":"2024-03-30T13:15:42.413284+0000","last_undegraded":"2024-03-30T13:15:42.413284+0000","last_fullsized":"2024-03-30T13:15:42.413284+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:14:50.629162+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:56.386997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":813,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-30T13:15:42.416457+0000","last_change":"2024-03-30T13:15:29.285894+0000","last_active":"2024-03-30T13:15:42.416457+0000","last_peered":"2024-03-30T13:15:42.416457+0000","last_clean":"2024-03-30T13:15:42.416457+0000","last_became_active":"2024-03-30T13:15:29.284839+0000","last_became_peered":"2024-03-30T13:15:29.284839+0000","last_unstale":"2024-03-30T13:15:42.416457+0000","last_undegraded":"2024-03-30T13:15:42.416457+0000","last_fullsized":"2024-03-30T13:15:42.416457+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:15:03.464473+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:15:03.464473+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:31.803707+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":39075780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":734,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-30T13:15:42.416394+0000","last_change":"2024-03-30T13:15:16.348851+0000","last_active":"2024-03-30T13:15:42.416394+0000","last_peered":"2024-03-30T13:15:42.416394+0000","last_clean":"2024-03-30T13:15:42.416394+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:15:42.416394+0000","last_undegraded":"2024-03-30T13:15:42.416394+0000","last_fullsized":"2024-03-30T13:15:42.416394+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:16.348578+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:15:16.348578+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:16.882619+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":107,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-30T13:15:41.700579+0000","last_change":"2024-03-30T13:15:21.424552+0000","last_active":"2024-03-30T13:15:41.700579+0000","last_peered":"2024-03-30T13:15:41.700579+0000","last_clean":"2024-03-30T13:15:41.700579+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:15:41.700579+0000","last_undegraded":"2024-03-30T13:15:41.700579+0000","last_fullsized":"2024-03-30T13:15:41.700579+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:15:21.424313+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:15:21.424313+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:49.667982+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1270,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-30T13:15:41.627090+0000","last_change":"2024-03-30T13:14:58.602202+0000","last_active":"2024-03-30T13:15:41.627090+0000","last_peered":"2024-03-30T13:15:41.627090+0000","last_clean":"2024-03-30T13:15:41.627090+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:15:41.627090+0000","last_undegraded":"2024-03-30T13:15:41.627090+0000","last_fullsized":"2024-03-30T13:15:41.627090+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:14:58.601881+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:14:58.601881+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:18.959306+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":803,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-30T13:15:42.416408+0000","last_change":"2024-03-30T13:15:31.305638+0000","last_active":"2024-03-30T13:15:42.416408+0000","last_peered":"2024-03-30T13:15:42.416408+0000","last_clean":"2024-03-30T13:15:42.416408+0000","last_became_active":"2024-03-30T13:15:29.283073+0000","last_became_peered":"2024-03-30T13:15:29.283073+0000","last_unstale":"2024-03-30T13:15:42.416408+0000","last_undegraded":"2024-03-30T13:15:42.416408+0000","last_fullsized":"2024-03-30T13:15:42.416408+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:15:31.305358+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:15:31.305358+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:47.066127+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":43916097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":539,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-30T13:15:44.442385+0000","last_change":"2024-03-30T13:14:39.618860+0000","last_active":"2024-03-30T13:15:44.442385+0000","last_peered":"2024-03-30T13:15:44.442385+0000","last_clean":"2024-03-30T13:15:44.442385+0000","last_became_active":"2024-03-30T13:14:38.233325+0000","last_became_peered":"2024-03-30T13:14:38.233325+0000","last_unstale":"2024-03-30T13:15:44.442385+0000","last_undegraded":"2024-03-30T13:15:44.442385+0000","last_fullsized":"2024-03-30T13:15:44.442385+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:14:39.618688+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:14:39.618688+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:00.662395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1378,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-30T13:15:41.627150+0000","last_change":"2024-03-30T13:15:36.454725+0000","last_active":"2024-03-30T13:15:41.627150+0000","last_peered":"2024-03-30T13:15:41.627150+0000","last_clean":"2024-03-30T13:15:41.627150+0000","last_became_active":"2024-03-30T13:14:38.238174+0000","last_became_peered":"2024-03-30T13:14:38.238174+0000","last_unstale":"2024-03-30T13:15:41.627150+0000","last_undegraded":"2024-03-30T13:15:41.627150+0000","last_fullsized":"2024-03-30T13:15:41.627150+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:36.454575+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:15:36.454575+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:40.210170+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":71059072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1049,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-30T13:15:42.413147+0000","last_change":"2024-03-30T13:15:29.284525+0000","last_active":"2024-03-30T13:15:42.413147+0000","last_peered":"2024-03-30T13:15:42.413147+0000","last_clean":"2024-03-30T13:15:42.413147+0000","last_became_active":"2024-03-30T13:15:29.282697+0000","last_became_peered":"2024-03-30T13:15:29.282697+0000","last_unstale":"2024-03-30T13:15:42.413147+0000","last_undegraded":"2024-03-30T13:15:42.413147+0000","last_fullsized":"2024-03-30T13:15:42.413147+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:14:49.668025+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:49.668025+0000","last_clean_scrub_stamp":"2024-03-30T13:14:49.668025+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:59.022759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1260,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-03-30T13:15:41.196582+0000","last_change":"2024-03-30T13:15:29.284976+0000","last_active":"2024-03-30T13:15:41.196582+0000","last_peered":"2024-03-30T13:15:41.196582+0000","last_clean":"2024-03-30T13:15:41.196582+0000","last_became_active":"2024-03-30T13:15:29.283557+0000","last_became_peered":"2024-03-30T13:15:29.283557+0000","last_unstale":"2024-03-30T13:15:41.196582+0000","last_undegraded":"2024-03-30T13:15:41.196582+0000","last_fullsized":"2024-03-30T13:15:41.196582+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:15:27.030418+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:15:27.030418+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:52.751419+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25479082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1354,"reported_epoch":548,"state":"active+clean","last_fresh":"2024-03-30T13:15:41.196557+0000","last_change":"2024-03-30T13:15:12.093216+0000","last_active":"2024-03-30T13:15:41.196557+0000","last_peered":"2024-03-30T13:15:41.196557+0000","last_clean":"2024-03-30T13:15:41.196557+0000","last_became_active":"2024-03-30T13:14:38.236424+0000","last_became_peered":"2024-03-30T13:14:38.236424+0000","last_unstale":"2024-03-30T13:15:41.196557+0000","last_undegraded":"2024-03-30T13:15:41.196557+0000","last_fullsized":"2024-03-30T13:15:41.196557+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:15:12.092921+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:34.616328+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":76975004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":120,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-30T13:15:41.700569+0000","last_change":"2024-03-30T13:15:14.320075+0000","last_active":"2024-03-30T13:15:41.700569+0000","last_peered":"2024-03-30T13:15:41.700569+0000","last_clean":"2024-03-30T13:15:41.700569+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:15:41.700569+0000","last_undegraded":"2024-03-30T13:15:41.700569+0000","last_fullsized":"2024-03-30T13:15:41.700569+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:15:14.319813+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:15:14.319813+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:34.069151+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":705,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-30T13:15:42.416375+0000","last_change":"2024-03-30T13:14:47.424410+0000","last_active":"2024-03-30T13:15:42.416375+0000","last_peered":"2024-03-30T13:15:42.416375+0000","last_clean":"2024-03-30T13:15:42.416375+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:15:42.416375+0000","last_undegraded":"2024-03-30T13:15:42.416375+0000","last_fullsized":"2024-03-30T13:15:42.416375+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:14:47.424173+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:14:47.424173+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:47.850501+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":104,"reported_epoch":549,"state":"active+clean","last_fresh":"2024-03-30T13:15:41.700579+0000","last_change":"2024-03-30T13:15:39.529282+0000","last_active":"2024-03-30T13:15:41.700579+0000","last_peered":"2024-03-30T13:15:41.700579+0000","last_clean":"2024-03-30T13:15:41.700579+0000","last_became_active":"2024-03-30T13:14:38.235512+0000","last_became_peered":"2024-03-30T13:14:38.235512+0000","last_unstale":"2024-03-30T13:15:41.700579+0000","last_undegraded":"2024-03-30T13:15:41.700579+0000","last_fullsized":"2024-03-30T13:15:41.700579+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:15:39.528997+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:15:39.528997+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:00.848645+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":490,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-30T13:15:44.442620+0000","last_change":"2024-03-30T13:15:10.510582+0000","last_active":"2024-03-30T13:15:44.442620+0000","last_peered":"2024-03-30T13:15:44.442620+0000","last_clean":"2024-03-30T13:15:44.442620+0000","last_became_active":"2024-03-30T13:14:38.231065+0000","last_became_peered":"2024-03-30T13:14:38.231065+0000","last_unstale":"2024-03-30T13:15:44.442620+0000","last_undegraded":"2024-03-30T13:15:44.442620+0000","last_fullsized":"2024-03-30T13:15:44.442620+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:10.510312+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:15:10.510312+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:36.027524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15680733,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":407160904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834189,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57932,"kb_used_data":21624,"kb_used_omap":5,"kb_used_meta":36282,"kb_avail":94313908,"statfs":{"total":96636764160,"available":96577441792,"internally_reserved":0,"allocated":22142976,"data_stored":18476063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":37152866},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157455,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64028,"kb_used_data":26380,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94307812,"statfs":{"total":96636764160,"available":96571199488,"internally_reserved":0,"allocated":27013120,"data_stored":23339506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":536,"seq":2302102470661,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55752,"kb_used_data":18108,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94316088,"statfs":{"total":96636764160,"available":96579674112,"internally_reserved":0,"allocated":18542592,"data_stored":14855376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10991,"internal_metadata":38524177},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509686,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48232,"kb_used_data":12888,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323608,"statfs":{"total":96636764160,"available":96587374592,"internally_reserved":0,"allocated":13197312,"data_stored":9503295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":461,"seq":1979979923492,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48912,"kb_used_data":12164,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94322928,"statfs":{"total":96636764160,"available":96586678272,"internally_reserved":0,"allocated":12455936,"data_stored":8761919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7692,"internal_metadata":37609972},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509687,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51248,"kb_used_data":20064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320592,"statfs":{"total":96636764160,"available":96584286208,"internally_reserved":0,"allocated":20545536,"data_stored":16851519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297081,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54216,"kb_used_data":16572,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94317624,"statfs":{"total":96636764160,"available":96581246976,"internally_reserved":0,"allocated":16969728,"data_stored":13296114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147395,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50720,"kb_used_data":19028,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321120,"statfs":{"total":96636764160,"available":96584826880,"internally_reserved":0,"allocated":19484672,"data_stored":15811058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:15:46.395 INFO:tasks.thrashosds.thrasher:Choosing PG 1.e with acting set [4, 7, 3] 2024-03-30T13:15:46.395 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [0, 2, 6, 3, 7, 1, 5, 4] 2024-03-30T13:15:46.395 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-30T13:15:46.395 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:15:46.463 INFO:tasks.ceph.osd.4:Stopped 2024-03-30T13:15:46.463 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 2, 6, 3, 7, 1, 5, 4] 2024-03-30T13:15:46.463 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 4 2024-03-30T13:15:46.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:46.577+0000 7f65a0a38640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:46.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:46.593+0000 7f65a0a38640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:46.773 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:15:46.773+0000 7fbb38f08640 -1 mon.a@0(leader).osd e552 definitely_dead 0 2024-03-30T13:15:47.456 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.4. 2024-03-30T13:15:47.485 INFO:tasks.thrashosds.thrasher:dead_osds=[4], live_osds=[0, 2, 6, 3, 7, 1, 5] 2024-03-30T13:15:47.485 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:15:47.485 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:15:47.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:47.597+0000 7f335344d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:47.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:15:47.613+0000 7f335344d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:15:47.786 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:15:47.786 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:15:47.803 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1153,"stamp":"2024-03-30T13:15:47.454554+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":407160904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":375436,"kb_used_data":128876,"kb_used_omap":43,"kb_used_meta":246420,"kb_avail":660227444,"statfs":{"total":676457349120,"available":676072902656,"internally_reserved":0,"allocated":131969024,"data_stored":106110905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44964,"internal_metadata":252334172},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.321607"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1290,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-30T13:15:44.875986+0000","last_change":"2024-03-30T13:15:31.536644+0000","last_active":"2024-03-30T13:15:44.875986+0000","last_peered":"2024-03-30T13:15:44.875986+0000","last_clean":"2024-03-30T13:15:44.875986+0000","last_became_active":"2024-03-30T13:15:29.285871+0000","last_became_peered":"2024-03-30T13:15:29.285871+0000","last_unstale":"2024-03-30T13:15:44.875986+0000","last_undegraded":"2024-03-30T13:15:44.875986+0000","last_fullsized":"2024-03-30T13:15:44.875986+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:31.536364+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:15:31.536364+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:45.858456+0000","scrub_duration":515,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":27529470,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1177,"reported_epoch":550,"state":"stale+active+clean","last_fresh":"2024-03-30T13:15:42.413284+0000","last_change":"2024-03-30T13:15:29.524842+0000","last_active":"2024-03-30T13:15:42.413284+0000","last_peered":"2024-03-30T13:15:42.413284+0000","last_clean":"2024-03-30T13:15:42.413284+0000","last_became_active":"2024-03-30T13:15:29.523936+0000","last_became_peered":"2024-03-30T13:15:29.523936+0000","last_unstale":"2024-03-30T13:15:47.454576+0000","last_undegraded":"2024-03-30T13:15:42.413284+0000","last_fullsized":"2024-03-30T13:15:42.413284+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:14:50.629162+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:56.386997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":813,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-30T13:15:42.416457+0000","last_change":"2024-03-30T13:15:29.285894+0000","last_active":"2024-03-30T13:15:42.416457+0000","last_peered":"2024-03-30T13:15:42.416457+0000","last_clean":"2024-03-30T13:15:42.416457+0000","last_became_active":"2024-03-30T13:15:29.284839+0000","last_became_peered":"2024-03-30T13:15:29.284839+0000","last_unstale":"2024-03-30T13:15:42.416457+0000","last_undegraded":"2024-03-30T13:15:42.416457+0000","last_fullsized":"2024-03-30T13:15:42.416457+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:15:03.464473+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:15:03.464473+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:31.803707+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":39075780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":734,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-30T13:15:42.416394+0000","last_change":"2024-03-30T13:15:16.348851+0000","last_active":"2024-03-30T13:15:42.416394+0000","last_peered":"2024-03-30T13:15:42.416394+0000","last_clean":"2024-03-30T13:15:42.416394+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:15:42.416394+0000","last_undegraded":"2024-03-30T13:15:42.416394+0000","last_fullsized":"2024-03-30T13:15:42.416394+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:16.348578+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:15:16.348578+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:16.882619+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":113,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-30T13:15:45.088202+0000","last_change":"2024-03-30T13:15:21.424552+0000","last_active":"2024-03-30T13:15:45.088202+0000","last_peered":"2024-03-30T13:15:45.088202+0000","last_clean":"2024-03-30T13:15:45.088202+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:15:45.088202+0000","last_undegraded":"2024-03-30T13:15:45.088202+0000","last_fullsized":"2024-03-30T13:15:45.088202+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:15:21.424313+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:15:21.424313+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:49.667982+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1276,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-30T13:15:45.085540+0000","last_change":"2024-03-30T13:14:58.602202+0000","last_active":"2024-03-30T13:15:45.085540+0000","last_peered":"2024-03-30T13:15:45.085540+0000","last_clean":"2024-03-30T13:15:45.085540+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:15:45.085540+0000","last_undegraded":"2024-03-30T13:15:45.085540+0000","last_fullsized":"2024-03-30T13:15:45.085540+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:14:58.601881+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:14:58.601881+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:18.959306+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":803,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-30T13:15:42.416408+0000","last_change":"2024-03-30T13:15:31.305638+0000","last_active":"2024-03-30T13:15:42.416408+0000","last_peered":"2024-03-30T13:15:42.416408+0000","last_clean":"2024-03-30T13:15:42.416408+0000","last_became_active":"2024-03-30T13:15:29.283073+0000","last_became_peered":"2024-03-30T13:15:29.283073+0000","last_unstale":"2024-03-30T13:15:42.416408+0000","last_undegraded":"2024-03-30T13:15:42.416408+0000","last_fullsized":"2024-03-30T13:15:42.416408+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:15:31.305358+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:15:31.305358+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:47.066127+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":43916097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":539,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-30T13:15:44.442385+0000","last_change":"2024-03-30T13:14:39.618860+0000","last_active":"2024-03-30T13:15:44.442385+0000","last_peered":"2024-03-30T13:15:44.442385+0000","last_clean":"2024-03-30T13:15:44.442385+0000","last_became_active":"2024-03-30T13:14:38.233325+0000","last_became_peered":"2024-03-30T13:14:38.233325+0000","last_unstale":"2024-03-30T13:15:44.442385+0000","last_undegraded":"2024-03-30T13:15:44.442385+0000","last_fullsized":"2024-03-30T13:15:44.442385+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:14:39.618688+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:14:39.618688+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:00.662395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1384,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-30T13:15:45.085330+0000","last_change":"2024-03-30T13:15:36.454725+0000","last_active":"2024-03-30T13:15:45.085330+0000","last_peered":"2024-03-30T13:15:45.085330+0000","last_clean":"2024-03-30T13:15:45.085330+0000","last_became_active":"2024-03-30T13:14:38.238174+0000","last_became_peered":"2024-03-30T13:14:38.238174+0000","last_unstale":"2024-03-30T13:15:45.085330+0000","last_undegraded":"2024-03-30T13:15:45.085330+0000","last_fullsized":"2024-03-30T13:15:45.085330+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:36.454575+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:15:36.454575+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:40.210170+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":71059072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1049,"reported_epoch":550,"state":"stale+active+clean","last_fresh":"2024-03-30T13:15:42.413147+0000","last_change":"2024-03-30T13:15:29.284525+0000","last_active":"2024-03-30T13:15:42.413147+0000","last_peered":"2024-03-30T13:15:42.413147+0000","last_clean":"2024-03-30T13:15:42.413147+0000","last_became_active":"2024-03-30T13:15:29.282697+0000","last_became_peered":"2024-03-30T13:15:29.282697+0000","last_unstale":"2024-03-30T13:15:47.454580+0000","last_undegraded":"2024-03-30T13:15:42.413147+0000","last_fullsized":"2024-03-30T13:15:42.413147+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:14:49.668025+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:49.668025+0000","last_clean_scrub_stamp":"2024-03-30T13:14:49.668025+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:59.022759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1267,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-30T13:15:44.875878+0000","last_change":"2024-03-30T13:15:29.284976+0000","last_active":"2024-03-30T13:15:44.875878+0000","last_peered":"2024-03-30T13:15:44.875878+0000","last_clean":"2024-03-30T13:15:44.875878+0000","last_became_active":"2024-03-30T13:15:29.283557+0000","last_became_peered":"2024-03-30T13:15:29.283557+0000","last_unstale":"2024-03-30T13:15:44.875878+0000","last_undegraded":"2024-03-30T13:15:44.875878+0000","last_fullsized":"2024-03-30T13:15:44.875878+0000","mapping_epoch":536,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":537,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:15:27.030418+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:15:27.030418+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:52.751419+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":25479082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1361,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-30T13:15:44.875977+0000","last_change":"2024-03-30T13:15:12.093216+0000","last_active":"2024-03-30T13:15:44.875977+0000","last_peered":"2024-03-30T13:15:44.875977+0000","last_clean":"2024-03-30T13:15:44.875977+0000","last_became_active":"2024-03-30T13:14:38.236424+0000","last_became_peered":"2024-03-30T13:14:38.236424+0000","last_unstale":"2024-03-30T13:15:44.875977+0000","last_undegraded":"2024-03-30T13:15:44.875977+0000","last_fullsized":"2024-03-30T13:15:44.875977+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:15:12.092921+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:34.616328+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":76975004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":126,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-30T13:15:45.088146+0000","last_change":"2024-03-30T13:15:14.320075+0000","last_active":"2024-03-30T13:15:45.088146+0000","last_peered":"2024-03-30T13:15:45.088146+0000","last_clean":"2024-03-30T13:15:45.088146+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:15:45.088146+0000","last_undegraded":"2024-03-30T13:15:45.088146+0000","last_fullsized":"2024-03-30T13:15:45.088146+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:15:14.319813+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:15:14.319813+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:34.069151+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":705,"reported_epoch":550,"state":"active+clean","last_fresh":"2024-03-30T13:15:42.416375+0000","last_change":"2024-03-30T13:14:47.424410+0000","last_active":"2024-03-30T13:15:42.416375+0000","last_peered":"2024-03-30T13:15:42.416375+0000","last_clean":"2024-03-30T13:15:42.416375+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:15:42.416375+0000","last_undegraded":"2024-03-30T13:15:42.416375+0000","last_fullsized":"2024-03-30T13:15:42.416375+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:14:47.424173+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:14:47.424173+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:15:47.850501+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":110,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-30T13:15:45.088214+0000","last_change":"2024-03-30T13:15:39.529282+0000","last_active":"2024-03-30T13:15:45.088214+0000","last_peered":"2024-03-30T13:15:45.088214+0000","last_clean":"2024-03-30T13:15:45.088214+0000","last_became_active":"2024-03-30T13:14:38.235512+0000","last_became_peered":"2024-03-30T13:14:38.235512+0000","last_unstale":"2024-03-30T13:15:45.088214+0000","last_undegraded":"2024-03-30T13:15:45.088214+0000","last_fullsized":"2024-03-30T13:15:45.088214+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:15:39.528997+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:15:39.528997+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:00.848645+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":490,"reported_epoch":552,"state":"active+clean","last_fresh":"2024-03-30T13:15:44.442620+0000","last_change":"2024-03-30T13:15:10.510582+0000","last_active":"2024-03-30T13:15:44.442620+0000","last_peered":"2024-03-30T13:15:44.442620+0000","last_clean":"2024-03-30T13:15:44.442620+0000","last_became_active":"2024-03-30T13:14:38.231065+0000","last_became_peered":"2024-03-30T13:14:38.231065+0000","last_unstale":"2024-03-30T13:15:44.442620+0000","last_undegraded":"2024-03-30T13:15:44.442620+0000","last_fullsized":"2024-03-30T13:15:44.442620+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:10.510312+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:15:10.510312+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:36.027524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15680733,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":407160904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834190,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57984,"kb_used_data":21684,"kb_used_omap":5,"kb_used_meta":36282,"kb_avail":94313856,"statfs":{"total":96636764160,"available":96577388544,"internally_reserved":0,"allocated":22204416,"data_stored":18503598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":37152866},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157456,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64072,"kb_used_data":26428,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94307768,"statfs":{"total":96636764160,"available":96571154432,"internally_reserved":0,"allocated":27062272,"data_stored":23361454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509686,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48232,"kb_used_data":12888,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323608,"statfs":{"total":96636764160,"available":96587374592,"internally_reserved":0,"allocated":13197312,"data_stored":9503295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":461,"seq":1979979923492,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48912,"kb_used_data":12164,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94322928,"statfs":{"total":96636764160,"available":96586678272,"internally_reserved":0,"allocated":12455936,"data_stored":8761919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7692,"internal_metadata":37609972},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509687,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51248,"kb_used_data":20064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320592,"statfs":{"total":96636764160,"available":96584286208,"internally_reserved":0,"allocated":20545536,"data_stored":16851519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297081,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54216,"kb_used_data":16572,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94317624,"statfs":{"total":96636764160,"available":96581246976,"internally_reserved":0,"allocated":16969728,"data_stored":13296114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147396,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50772,"kb_used_data":19076,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94321068,"statfs":{"total":96636764160,"available":96584773632,"internally_reserved":0,"allocated":19533824,"data_stored":15833006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:15:47.805 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:16:12.806 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:16:12.807 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:16:12.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:12.932+0000 7f7e29cea640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:12.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:12.948+0000 7f7e29cea640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:13.125 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:16:13.125 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:16:13.143 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1170,"stamp":"2024-03-30T13:16:12.141483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":446942545,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":388064,"kb_used_data":141520,"kb_used_omap":43,"kb_used_meta":246420,"kb_avail":660214816,"statfs":{"total":676457349120,"available":676059971584,"internally_reserved":0,"allocated":144916480,"data_stored":118815611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44964,"internal_metadata":252334172},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.934406"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1312,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-03-30T13:16:10.573977+0000","last_change":"2024-03-30T13:15:48.531055+0000","last_active":"2024-03-30T13:16:10.573977+0000","last_peered":"2024-03-30T13:16:10.573977+0000","last_clean":"2024-03-30T13:16:10.573977+0000","last_became_active":"2024-03-30T13:15:48.478873+0000","last_became_peered":"2024-03-30T13:15:48.478873+0000","last_unstale":"2024-03-30T13:16:10.573977+0000","last_undegraded":"2024-03-30T13:16:10.573977+0000","last_fullsized":"2024-03-30T13:16:10.573977+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":554,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:31.536364+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:15:31.536364+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:33.074185+0000","scrub_duration":515,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":32117715,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":27,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-30T13:16:02.397952+0000","last_change":"2024-03-30T13:16:02.398081+0000","last_active":"2024-03-30T13:16:02.397952+0000","last_peered":"2024-03-30T13:16:02.397952+0000","last_clean":"2024-03-30T13:16:02.397952+0000","last_became_active":"2024-03-30T13:15:48.486333+0000","last_became_peered":"2024-03-30T13:15:48.486333+0000","last_unstale":"2024-03-30T13:16:02.397952+0000","last_undegraded":"2024-03-30T13:16:02.397952+0000","last_fullsized":"2024-03-30T13:16:02.397952+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":554,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:16:02.397699+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:16:02.397699+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:29.185626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":847,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-30T13:16:04.484260+0000","last_change":"2024-03-30T13:16:04.484341+0000","last_active":"2024-03-30T13:16:04.484260+0000","last_peered":"2024-03-30T13:16:04.484260+0000","last_clean":"2024-03-30T13:16:04.484260+0000","last_became_active":"2024-03-30T13:15:48.481182+0000","last_became_peered":"2024-03-30T13:15:48.481182+0000","last_unstale":"2024-03-30T13:16:04.484260+0000","last_undegraded":"2024-03-30T13:16:04.484260+0000","last_fullsized":"2024-03-30T13:16:04.484260+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":554,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:16:04.484050+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:16:04.484050+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:08.091622+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":46890936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":746,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-30T13:16:02.372673+0000","last_change":"2024-03-30T13:15:16.348851+0000","last_active":"2024-03-30T13:16:02.372673+0000","last_peered":"2024-03-30T13:16:02.372673+0000","last_clean":"2024-03-30T13:16:02.372673+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:16:02.372673+0000","last_undegraded":"2024-03-30T13:16:02.372673+0000","last_fullsized":"2024-03-30T13:16:02.372673+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:16.348578+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:15:16.348578+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:16.882619+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":121,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-30T13:16:02.209470+0000","last_change":"2024-03-30T13:15:21.424552+0000","last_active":"2024-03-30T13:16:02.209470+0000","last_peered":"2024-03-30T13:16:02.209470+0000","last_clean":"2024-03-30T13:16:02.209470+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:16:02.209470+0000","last_undegraded":"2024-03-30T13:16:02.209470+0000","last_fullsized":"2024-03-30T13:16:02.209470+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:15:21.424313+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:15:21.424313+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:49.667982+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1284,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-30T13:16:02.370546+0000","last_change":"2024-03-30T13:14:58.602202+0000","last_active":"2024-03-30T13:16:02.370546+0000","last_peered":"2024-03-30T13:16:02.370546+0000","last_clean":"2024-03-30T13:16:02.370546+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:16:02.370546+0000","last_undegraded":"2024-03-30T13:16:02.370546+0000","last_fullsized":"2024-03-30T13:16:02.370546+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:14:58.601881+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:14:58.601881+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:18.959306+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":835,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-30T13:16:02.372860+0000","last_change":"2024-03-30T13:15:49.184894+0000","last_active":"2024-03-30T13:16:02.372860+0000","last_peered":"2024-03-30T13:16:02.372860+0000","last_clean":"2024-03-30T13:16:02.372860+0000","last_became_active":"2024-03-30T13:15:48.481143+0000","last_became_peered":"2024-03-30T13:15:48.481143+0000","last_unstale":"2024-03-30T13:16:02.372860+0000","last_undegraded":"2024-03-30T13:16:02.372860+0000","last_fullsized":"2024-03-30T13:16:02.372860+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":554,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:15:31.305358+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:15:31.305358+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:52.461575+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58554796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":557,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-03-30T13:16:10.234530+0000","last_change":"2024-03-30T13:16:09.352505+0000","last_active":"2024-03-30T13:16:10.234530+0000","last_peered":"2024-03-30T13:16:10.234530+0000","last_clean":"2024-03-30T13:16:10.234530+0000","last_became_active":"2024-03-30T13:14:38.233325+0000","last_became_peered":"2024-03-30T13:14:38.233325+0000","last_unstale":"2024-03-30T13:16:10.234530+0000","last_undegraded":"2024-03-30T13:16:10.234530+0000","last_fullsized":"2024-03-30T13:16:10.234530+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:16:09.352215+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:16:09.352215+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:32.702881+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1392,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-30T13:16:02.370369+0000","last_change":"2024-03-30T13:15:36.454725+0000","last_active":"2024-03-30T13:16:02.370369+0000","last_peered":"2024-03-30T13:16:02.370369+0000","last_clean":"2024-03-30T13:16:02.370369+0000","last_became_active":"2024-03-30T13:14:38.238174+0000","last_became_peered":"2024-03-30T13:14:38.238174+0000","last_unstale":"2024-03-30T13:16:02.370369+0000","last_undegraded":"2024-03-30T13:16:02.370369+0000","last_fullsized":"2024-03-30T13:16:02.370369+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:36.454575+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:15:36.454575+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:40.210170+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":71059072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":19,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-30T13:16:02.370171+0000","last_change":"2024-03-30T13:15:48.512633+0000","last_active":"2024-03-30T13:16:02.370171+0000","last_peered":"2024-03-30T13:16:02.370171+0000","last_clean":"2024-03-30T13:16:02.370171+0000","last_became_active":"2024-03-30T13:15:48.479171+0000","last_became_peered":"2024-03-30T13:15:48.479171+0000","last_unstale":"2024-03-30T13:16:02.370171+0000","last_undegraded":"2024-03-30T13:16:02.370171+0000","last_fullsized":"2024-03-30T13:16:02.370171+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":554,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:14:49.668025+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:49.668025+0000","last_clean_scrub_stamp":"2024-03-30T13:14:49.668025+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:15.724248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1296,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-03-30T13:16:10.573807+0000","last_change":"2024-03-30T13:15:48.593594+0000","last_active":"2024-03-30T13:16:10.573807+0000","last_peered":"2024-03-30T13:16:10.573807+0000","last_clean":"2024-03-30T13:16:10.573807+0000","last_became_active":"2024-03-30T13:15:48.491891+0000","last_became_peered":"2024-03-30T13:15:48.491891+0000","last_unstale":"2024-03-30T13:16:10.573807+0000","last_undegraded":"2024-03-30T13:16:10.573807+0000","last_fullsized":"2024-03-30T13:16:10.573807+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":554,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:15:27.030418+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:15:27.030418+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:49.457410+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":38218623,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1372,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-03-30T13:16:10.573683+0000","last_change":"2024-03-30T13:15:12.093216+0000","last_active":"2024-03-30T13:16:10.573683+0000","last_peered":"2024-03-30T13:16:10.573683+0000","last_clean":"2024-03-30T13:16:10.573683+0000","last_became_active":"2024-03-30T13:14:38.236424+0000","last_became_peered":"2024-03-30T13:14:38.236424+0000","last_unstale":"2024-03-30T13:16:10.573683+0000","last_undegraded":"2024-03-30T13:16:10.573683+0000","last_fullsized":"2024-03-30T13:16:10.573683+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:15:12.092921+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:34.616328+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":76975004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":134,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-30T13:16:02.209504+0000","last_change":"2024-03-30T13:15:14.320075+0000","last_active":"2024-03-30T13:16:02.209504+0000","last_peered":"2024-03-30T13:16:02.209504+0000","last_clean":"2024-03-30T13:16:02.209504+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:16:02.209504+0000","last_undegraded":"2024-03-30T13:16:02.209504+0000","last_fullsized":"2024-03-30T13:16:02.209504+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:15:14.319813+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:15:14.319813+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:34.069151+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":725,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-30T13:16:02.372677+0000","last_change":"2024-03-30T13:15:52.423713+0000","last_active":"2024-03-30T13:16:02.372677+0000","last_peered":"2024-03-30T13:16:02.372677+0000","last_clean":"2024-03-30T13:16:02.372677+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:16:02.372677+0000","last_undegraded":"2024-03-30T13:16:02.372677+0000","last_fullsized":"2024-03-30T13:16:02.372677+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:15:52.423297+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:15:52.423297+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:08.679223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":118,"reported_epoch":556,"state":"active+clean","last_fresh":"2024-03-30T13:16:02.209558+0000","last_change":"2024-03-30T13:15:39.529282+0000","last_active":"2024-03-30T13:16:02.209558+0000","last_peered":"2024-03-30T13:16:02.209558+0000","last_clean":"2024-03-30T13:16:02.209558+0000","last_became_active":"2024-03-30T13:14:38.235512+0000","last_became_peered":"2024-03-30T13:14:38.235512+0000","last_unstale":"2024-03-30T13:16:02.209558+0000","last_undegraded":"2024-03-30T13:16:02.209558+0000","last_fullsized":"2024-03-30T13:16:02.209558+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:15:39.528997+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:15:39.528997+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:00.848645+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":500,"reported_epoch":557,"state":"active+clean","last_fresh":"2024-03-30T13:16:10.234324+0000","last_change":"2024-03-30T13:15:10.510582+0000","last_active":"2024-03-30T13:16:10.234324+0000","last_peered":"2024-03-30T13:16:10.234324+0000","last_clean":"2024-03-30T13:16:10.234324+0000","last_became_active":"2024-03-30T13:14:38.231065+0000","last_became_peered":"2024-03-30T13:14:38.231065+0000","last_unstale":"2024-03-30T13:16:10.234324+0000","last_undegraded":"2024-03-30T13:16:10.234324+0000","last_fullsized":"2024-03-30T13:16:10.234324+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:10.510312+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:15:10.510312+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:36.027524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15680733,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":446942545,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":109256704,"data_stored":109256704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834195,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58044,"kb_used_data":21744,"kb_used_omap":5,"kb_used_meta":36282,"kb_avail":94313796,"statfs":{"total":96636764160,"available":96577327104,"internally_reserved":0,"allocated":22265856,"data_stored":18531367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":37152866},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157460,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64120,"kb_used_data":26476,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94307720,"statfs":{"total":96636764160,"available":96571105280,"internally_reserved":0,"allocated":27111424,"data_stored":23383694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509691,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55044,"kb_used_data":19704,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94316796,"statfs":{"total":96636764160,"available":96580399104,"internally_reserved":0,"allocated":20176896,"data_stored":16449166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":461,"seq":1979979923498,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50064,"kb_used_data":13316,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94321776,"statfs":{"total":96636764160,"available":96585498624,"internally_reserved":0,"allocated":13635584,"data_stored":9907854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7692,"internal_metadata":37609972},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509692,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51808,"kb_used_data":20628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320032,"statfs":{"total":96636764160,"available":96583712768,"internally_reserved":0,"allocated":21123072,"data_stored":17395342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297086,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54308,"kb_used_data":16668,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94317532,"statfs":{"total":96636764160,"available":96581152768,"internally_reserved":0,"allocated":17068032,"data_stored":13340302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147400,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54676,"kb_used_data":22984,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94317164,"statfs":{"total":96636764160,"available":96580775936,"internally_reserved":0,"allocated":23535616,"data_stored":19807886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14340096,"data_stored":14340096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6852608,"data_stored":6852608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13393920,"data_stored":13393920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16752640,"data_stored":16752640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:16:13.144 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:16:13.144 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:16:13.144 INFO:tasks.thrashosds.thrasher:Reviving osd 4 2024-03-30T13:16:13.144 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-30T13:16:13.145 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-30T13:16:13.147 INFO:tasks.ceph.osd.4:Started 2024-03-30T13:16:13.147 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-30T13:16:13.214 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:16:13.212+0000 7febaed837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:13.222 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:16:13.220+0000 7febaed837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:13.222 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:16:13.220+0000 7febaed837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:13.276 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T13:16:13.277 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T13:16:13.277 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T13:16:13.277 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T13:16:13.277 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T13:16:13.277 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T13:16:13.277 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-03-30T13:16:13.277 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-03-30T13:16:13.277 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-03-30T13:16:13.277 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-03-30T13:16:13.277 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-03-30T13:16:13.277 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T13:16:13.278 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:16:13.279 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-03-30T13:16:13.772 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:16:13.772+0000 7febaed837c0 -1 Falling back to public interface 2024-03-30T13:16:17.535 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:16:17.536+0000 7febaed837c0 -1 osd.4 552 log_to_monitors true 2024-03-30T13:16:18.280 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-30T13:16:18.331 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:16:18.332+0000 7feba6b58640 -1 osd.4 552 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:16:18.405 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T13:16:18.406 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T13:16:18.406 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T13:16:18.406 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T13:16:18.419 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:16:18.419 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-03-30T13:16:18.419 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 4 2024-03-30T13:16:18.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:18.540+0000 7f951f054640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:18.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:18.556+0000 7f951f054640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:18.741 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:16:18.736+0000 7fbb38f08640 -1 mon.a@0(leader).osd e558 definitely_dead 0 2024-03-30T13:16:19.290 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.4. 2024-03-30T13:16:19.319 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-03-30T13:16:19.319 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:16:19.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:19.436+0000 7f2ff33bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:19.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:19.452+0000 7f2ff33bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:20.303 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:16:20.331 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:16:20.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:20.444+0000 7faf01302640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:20.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:20.460+0000 7faf01302640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:21.305 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:16:21.335 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:16:21.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:21.452+0000 7f2827568640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:21.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:21.468+0000 7f2827568640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:22.322 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:16:22.345 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:16:22.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:22.464+0000 7f49d61c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:22.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:22.480+0000 7f49d61c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:23.326 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:16:23.355 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:16:23.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:23.472+0000 7f7b5288e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:23.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:23.488+0000 7f7b5288e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:24.346 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:16:24.374 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:16:24.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:24.496+0000 7ff92c626640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:24.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:24.512+0000 7ff92c626640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:25.362 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:16:25.386 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:16:25.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:25.504+0000 7f0956b0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:25.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:25.520+0000 7f0956b0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:26.374 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:16:26.401 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:16:26.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:26.520+0000 7ffb55936640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:26.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:26.536+0000 7ffb55936640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:27.395 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:16:27.423 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:16:27.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:27.544+0000 7f9f7a996640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:27.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:27.564+0000 7f9f7a996640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:28.390 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:16:28.418 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:16:28.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:28.528+0000 7fbb83e55640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:28.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:28.544+0000 7fbb83e55640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:29.405 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:16:29.429 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:16:29.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:29.544+0000 7ff7b7989640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:29.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:29.556+0000 7ff7b7989640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:30.421 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:16:30.447 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:16:30.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:30.564+0000 7f4e28c3e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:30.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:30.584+0000 7f4e28c3e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:31.418 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:16:31.443 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:16:31.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:31.560+0000 7f62fe723640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:31.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:31.576+0000 7f62fe723640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:32.435 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:16:32.459 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:16:32.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:32.571+0000 7f0a1e612640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:32.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:32.587+0000 7f0a1e612640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:33.437 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:16:33.469 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:16:33.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:33.587+0000 7f6281f83640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:33.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:33.603+0000 7f6281f83640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:34.456 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:16:34.482 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:16:34.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:34.599+0000 7f2e35c08640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:34.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:34.615+0000 7f2e35c08640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:35.473 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:16:35.499 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:16:35.499 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:16:35.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:35.615+0000 7f0d6acc4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:35.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:35.627+0000 7f0d6acc4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:35.813 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:16:35.814 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:16:35.831 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1199,"stamp":"2024-03-30T13:16:35.473954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":446942545,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":433216,"kb_used_data":149000,"kb_used_omap":56,"kb_used_meta":284039,"kb_avail":754541504,"statfs":{"total":773094113280,"available":772650500096,"internally_reserved":0,"allocated":152576000,"data_stored":121903641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57603,"internal_metadata":290856701},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.049400"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1342,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-30T13:16:30.610334+0000","last_change":"2024-03-30T13:16:20.317770+0000","last_active":"2024-03-30T13:16:30.610334+0000","last_peered":"2024-03-30T13:16:30.610334+0000","last_clean":"2024-03-30T13:16:30.610334+0000","last_became_active":"2024-03-30T13:16:20.316817+0000","last_became_peered":"2024-03-30T13:16:20.316817+0000","last_unstale":"2024-03-30T13:16:30.610334+0000","last_undegraded":"2024-03-30T13:16:30.610334+0000","last_fullsized":"2024-03-30T13:16:30.610334+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:31.536364+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:15:31.536364+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:48.846664+0000","scrub_duration":515,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":32117715,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1190,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-03-30T13:16:33.446532+0000","last_change":"2024-03-30T13:16:20.315116+0000","last_active":"2024-03-30T13:16:33.446532+0000","last_peered":"2024-03-30T13:16:33.446532+0000","last_clean":"2024-03-30T13:16:33.446532+0000","last_became_active":"2024-03-30T13:16:20.313939+0000","last_became_peered":"2024-03-30T13:16:20.313939+0000","last_unstale":"2024-03-30T13:16:33.446532+0000","last_undegraded":"2024-03-30T13:16:33.446532+0000","last_fullsized":"2024-03-30T13:16:33.446532+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:16:02.397699+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:16:02.397699+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:29.215054+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":884,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-30T13:16:32.446883+0000","last_change":"2024-03-30T13:16:20.312577+0000","last_active":"2024-03-30T13:16:32.446883+0000","last_peered":"2024-03-30T13:16:32.446883+0000","last_clean":"2024-03-30T13:16:32.446883+0000","last_became_active":"2024-03-30T13:16:20.311258+0000","last_became_peered":"2024-03-30T13:16:20.311258+0000","last_unstale":"2024-03-30T13:16:32.446883+0000","last_undegraded":"2024-03-30T13:16:32.446883+0000","last_fullsized":"2024-03-30T13:16:32.446883+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:16:04.484050+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:16:04.484050+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:18.836317+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":46890936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":785,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-30T13:16:32.446828+0000","last_change":"2024-03-30T13:16:20.586918+0000","last_active":"2024-03-30T13:16:32.446828+0000","last_peered":"2024-03-30T13:16:32.446828+0000","last_clean":"2024-03-30T13:16:32.446828+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:16:32.446828+0000","last_undegraded":"2024-03-30T13:16:32.446828+0000","last_fullsized":"2024-03-30T13:16:32.446828+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:20.586668+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:16:20.586668+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:50.224020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":147,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-30T13:16:30.424286+0000","last_change":"2024-03-30T13:15:21.424552+0000","last_active":"2024-03-30T13:16:30.424286+0000","last_peered":"2024-03-30T13:16:30.424286+0000","last_clean":"2024-03-30T13:16:30.424286+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:16:30.424286+0000","last_undegraded":"2024-03-30T13:16:30.424286+0000","last_fullsized":"2024-03-30T13:16:30.424286+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:15:21.424313+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:15:21.424313+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:49.667982+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1320,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-30T13:16:30.435851+0000","last_change":"2024-03-30T13:16:19.587276+0000","last_active":"2024-03-30T13:16:30.435851+0000","last_peered":"2024-03-30T13:16:30.435851+0000","last_clean":"2024-03-30T13:16:30.435851+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:16:30.435851+0000","last_undegraded":"2024-03-30T13:16:30.435851+0000","last_fullsized":"2024-03-30T13:16:30.435851+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:16:19.587123+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:16:19.587123+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:25.576107+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":872,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-30T13:16:32.446858+0000","last_change":"2024-03-30T13:16:20.314213+0000","last_active":"2024-03-30T13:16:32.446858+0000","last_peered":"2024-03-30T13:16:32.446858+0000","last_clean":"2024-03-30T13:16:32.446858+0000","last_became_active":"2024-03-30T13:16:20.312300+0000","last_became_peered":"2024-03-30T13:16:20.312300+0000","last_unstale":"2024-03-30T13:16:32.446858+0000","last_undegraded":"2024-03-30T13:16:32.446858+0000","last_fullsized":"2024-03-30T13:16:32.446858+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:15:31.305358+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:15:31.305358+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:57.539681+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58554796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":591,"reported_epoch":574,"state":"active+clean","last_fresh":"2024-03-30T13:16:34.454358+0000","last_change":"2024-03-30T13:16:09.352505+0000","last_active":"2024-03-30T13:16:34.454358+0000","last_peered":"2024-03-30T13:16:34.454358+0000","last_clean":"2024-03-30T13:16:34.454358+0000","last_became_active":"2024-03-30T13:14:38.233325+0000","last_became_peered":"2024-03-30T13:14:38.233325+0000","last_unstale":"2024-03-30T13:16:34.454358+0000","last_undegraded":"2024-03-30T13:16:34.454358+0000","last_fullsized":"2024-03-30T13:16:34.454358+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:16:09.352215+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:16:09.352215+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:32.702881+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1421,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-30T13:16:30.435963+0000","last_change":"2024-03-30T13:15:36.454725+0000","last_active":"2024-03-30T13:16:30.435963+0000","last_peered":"2024-03-30T13:16:30.435963+0000","last_clean":"2024-03-30T13:16:30.435963+0000","last_became_active":"2024-03-30T13:14:38.238174+0000","last_became_peered":"2024-03-30T13:14:38.238174+0000","last_unstale":"2024-03-30T13:16:30.435963+0000","last_undegraded":"2024-03-30T13:16:30.435963+0000","last_fullsized":"2024-03-30T13:16:30.435963+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:36.454575+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:15:36.454575+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:40.210170+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":71059072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1062,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-03-30T13:16:33.446423+0000","last_change":"2024-03-30T13:16:20.318505+0000","last_active":"2024-03-30T13:16:33.446423+0000","last_peered":"2024-03-30T13:16:33.446423+0000","last_clean":"2024-03-30T13:16:33.446423+0000","last_became_active":"2024-03-30T13:16:20.315944+0000","last_became_peered":"2024-03-30T13:16:20.315944+0000","last_unstale":"2024-03-30T13:16:33.446423+0000","last_undegraded":"2024-03-30T13:16:33.446423+0000","last_fullsized":"2024-03-30T13:16:33.446423+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_clean_scrub_stamp":"2024-03-30T13:16:16.623820+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:35.532032+0000","scrub_duration":659,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1326,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-30T13:16:30.609908+0000","last_change":"2024-03-30T13:16:20.314679+0000","last_active":"2024-03-30T13:16:30.609908+0000","last_peered":"2024-03-30T13:16:30.609908+0000","last_clean":"2024-03-30T13:16:30.609908+0000","last_became_active":"2024-03-30T13:16:20.313826+0000","last_became_peered":"2024-03-30T13:16:20.313826+0000","last_unstale":"2024-03-30T13:16:30.609908+0000","last_undegraded":"2024-03-30T13:16:30.609908+0000","last_fullsized":"2024-03-30T13:16:30.609908+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:15:27.030418+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:15:27.030418+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:31.173280+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":38218623,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1396,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-30T13:16:30.610149+0000","last_change":"2024-03-30T13:15:12.093216+0000","last_active":"2024-03-30T13:16:30.610149+0000","last_peered":"2024-03-30T13:16:30.610149+0000","last_clean":"2024-03-30T13:16:30.610149+0000","last_became_active":"2024-03-30T13:14:38.236424+0000","last_became_peered":"2024-03-30T13:14:38.236424+0000","last_unstale":"2024-03-30T13:16:30.610149+0000","last_undegraded":"2024-03-30T13:16:30.610149+0000","last_fullsized":"2024-03-30T13:16:30.610149+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:15:12.092921+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:34.616328+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":76975004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":160,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-30T13:16:30.424257+0000","last_change":"2024-03-30T13:15:14.320075+0000","last_active":"2024-03-30T13:16:30.424257+0000","last_peered":"2024-03-30T13:16:30.424257+0000","last_clean":"2024-03-30T13:16:30.424257+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:16:30.424257+0000","last_undegraded":"2024-03-30T13:16:30.424257+0000","last_fullsized":"2024-03-30T13:16:30.424257+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:15:14.319813+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:15:14.319813+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:34.069151+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":756,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-30T13:16:32.447050+0000","last_change":"2024-03-30T13:15:52.423713+0000","last_active":"2024-03-30T13:16:32.447050+0000","last_peered":"2024-03-30T13:16:32.447050+0000","last_clean":"2024-03-30T13:16:32.447050+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:16:32.447050+0000","last_undegraded":"2024-03-30T13:16:32.447050+0000","last_fullsized":"2024-03-30T13:16:32.447050+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:15:52.423297+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:15:52.423297+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:08.679223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":144,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-30T13:16:30.424306+0000","last_change":"2024-03-30T13:15:39.529282+0000","last_active":"2024-03-30T13:16:30.424306+0000","last_peered":"2024-03-30T13:16:30.424306+0000","last_clean":"2024-03-30T13:16:30.424306+0000","last_became_active":"2024-03-30T13:14:38.235512+0000","last_became_peered":"2024-03-30T13:14:38.235512+0000","last_unstale":"2024-03-30T13:16:30.424306+0000","last_undegraded":"2024-03-30T13:16:30.424306+0000","last_fullsized":"2024-03-30T13:16:30.424306+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:15:39.528997+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:15:39.528997+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:00.848645+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":534,"reported_epoch":574,"state":"active+clean","last_fresh":"2024-03-30T13:16:34.454380+0000","last_change":"2024-03-30T13:15:10.510582+0000","last_active":"2024-03-30T13:16:34.454380+0000","last_peered":"2024-03-30T13:16:34.454380+0000","last_clean":"2024-03-30T13:16:34.454380+0000","last_became_active":"2024-03-30T13:14:38.231065+0000","last_became_peered":"2024-03-30T13:14:38.231065+0000","last_unstale":"2024-03-30T13:16:34.454380+0000","last_undegraded":"2024-03-30T13:16:34.454380+0000","last_fullsized":"2024-03-30T13:16:34.454380+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:10.510312+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:15:10.510312+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:36.027524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15680733,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":446942545,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834199,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58204,"kb_used_data":21900,"kb_used_omap":5,"kb_used_meta":36282,"kb_avail":94313636,"statfs":{"total":96636764160,"available":96577163264,"internally_reserved":0,"allocated":22425600,"data_stored":18604307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":37152866},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157465,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64288,"kb_used_data":26644,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94307552,"statfs":{"total":96636764160,"available":96570933248,"internally_reserved":0,"allocated":27283456,"data_stored":23462163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":559,"seq":2400886718469,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56028,"kb_used_data":18384,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94315812,"statfs":{"total":96636764160,"available":96579391488,"internally_reserved":0,"allocated":18825216,"data_stored":14983520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12639,"internal_metadata":38522529},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509696,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48496,"kb_used_data":13152,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323344,"statfs":{"total":96636764160,"available":96587104256,"internally_reserved":0,"allocated":13467648,"data_stored":9625952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":461,"seq":1979979923503,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49188,"kb_used_data":12440,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94322652,"statfs":{"total":96636764160,"available":96586395648,"internally_reserved":0,"allocated":12738560,"data_stored":8890063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7692,"internal_metadata":37609972},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509697,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51516,"kb_used_data":20328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320324,"statfs":{"total":96636764160,"available":96584011776,"internally_reserved":0,"allocated":20815872,"data_stored":16974176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297091,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54508,"kb_used_data":16860,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94317332,"statfs":{"total":96636764160,"available":96580947968,"internally_reserved":0,"allocated":17264640,"data_stored":13429745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147405,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50988,"kb_used_data":19292,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94320852,"statfs":{"total":96636764160,"available":96584552448,"internally_reserved":0,"allocated":19755008,"data_stored":15933715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:16:35.832 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:16:35.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:35.947+0000 7f717ec14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:35.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:35.963+0000 7f717ec14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:36.141 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:16:36.142 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:16:36.159 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1199,"stamp":"2024-03-30T13:16:35.473954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":446942545,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":433216,"kb_used_data":149000,"kb_used_omap":56,"kb_used_meta":284039,"kb_avail":754541504,"statfs":{"total":773094113280,"available":772650500096,"internally_reserved":0,"allocated":152576000,"data_stored":121903641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57603,"internal_metadata":290856701},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.049400"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1342,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-30T13:16:30.610334+0000","last_change":"2024-03-30T13:16:20.317770+0000","last_active":"2024-03-30T13:16:30.610334+0000","last_peered":"2024-03-30T13:16:30.610334+0000","last_clean":"2024-03-30T13:16:30.610334+0000","last_became_active":"2024-03-30T13:16:20.316817+0000","last_became_peered":"2024-03-30T13:16:20.316817+0000","last_unstale":"2024-03-30T13:16:30.610334+0000","last_undegraded":"2024-03-30T13:16:30.610334+0000","last_fullsized":"2024-03-30T13:16:30.610334+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:31.536364+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:15:31.536364+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:48.846664+0000","scrub_duration":515,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":32117715,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1190,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-03-30T13:16:33.446532+0000","last_change":"2024-03-30T13:16:20.315116+0000","last_active":"2024-03-30T13:16:33.446532+0000","last_peered":"2024-03-30T13:16:33.446532+0000","last_clean":"2024-03-30T13:16:33.446532+0000","last_became_active":"2024-03-30T13:16:20.313939+0000","last_became_peered":"2024-03-30T13:16:20.313939+0000","last_unstale":"2024-03-30T13:16:33.446532+0000","last_undegraded":"2024-03-30T13:16:33.446532+0000","last_fullsized":"2024-03-30T13:16:33.446532+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:16:02.397699+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:16:02.397699+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:29.215054+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":884,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-30T13:16:32.446883+0000","last_change":"2024-03-30T13:16:20.312577+0000","last_active":"2024-03-30T13:16:32.446883+0000","last_peered":"2024-03-30T13:16:32.446883+0000","last_clean":"2024-03-30T13:16:32.446883+0000","last_became_active":"2024-03-30T13:16:20.311258+0000","last_became_peered":"2024-03-30T13:16:20.311258+0000","last_unstale":"2024-03-30T13:16:32.446883+0000","last_undegraded":"2024-03-30T13:16:32.446883+0000","last_fullsized":"2024-03-30T13:16:32.446883+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:16:04.484050+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:16:04.484050+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:18.836317+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":46890936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":785,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-30T13:16:32.446828+0000","last_change":"2024-03-30T13:16:20.586918+0000","last_active":"2024-03-30T13:16:32.446828+0000","last_peered":"2024-03-30T13:16:32.446828+0000","last_clean":"2024-03-30T13:16:32.446828+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:16:32.446828+0000","last_undegraded":"2024-03-30T13:16:32.446828+0000","last_fullsized":"2024-03-30T13:16:32.446828+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:20.586668+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:16:20.586668+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:50.224020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":147,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-30T13:16:30.424286+0000","last_change":"2024-03-30T13:15:21.424552+0000","last_active":"2024-03-30T13:16:30.424286+0000","last_peered":"2024-03-30T13:16:30.424286+0000","last_clean":"2024-03-30T13:16:30.424286+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:16:30.424286+0000","last_undegraded":"2024-03-30T13:16:30.424286+0000","last_fullsized":"2024-03-30T13:16:30.424286+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:15:21.424313+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:15:21.424313+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:49.667982+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1320,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-30T13:16:30.435851+0000","last_change":"2024-03-30T13:16:19.587276+0000","last_active":"2024-03-30T13:16:30.435851+0000","last_peered":"2024-03-30T13:16:30.435851+0000","last_clean":"2024-03-30T13:16:30.435851+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:16:30.435851+0000","last_undegraded":"2024-03-30T13:16:30.435851+0000","last_fullsized":"2024-03-30T13:16:30.435851+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:16:19.587123+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:16:19.587123+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:25.576107+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":872,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-30T13:16:32.446858+0000","last_change":"2024-03-30T13:16:20.314213+0000","last_active":"2024-03-30T13:16:32.446858+0000","last_peered":"2024-03-30T13:16:32.446858+0000","last_clean":"2024-03-30T13:16:32.446858+0000","last_became_active":"2024-03-30T13:16:20.312300+0000","last_became_peered":"2024-03-30T13:16:20.312300+0000","last_unstale":"2024-03-30T13:16:32.446858+0000","last_undegraded":"2024-03-30T13:16:32.446858+0000","last_fullsized":"2024-03-30T13:16:32.446858+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:15:31.305358+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:15:31.305358+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:57.539681+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58554796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":591,"reported_epoch":574,"state":"active+clean","last_fresh":"2024-03-30T13:16:34.454358+0000","last_change":"2024-03-30T13:16:09.352505+0000","last_active":"2024-03-30T13:16:34.454358+0000","last_peered":"2024-03-30T13:16:34.454358+0000","last_clean":"2024-03-30T13:16:34.454358+0000","last_became_active":"2024-03-30T13:14:38.233325+0000","last_became_peered":"2024-03-30T13:14:38.233325+0000","last_unstale":"2024-03-30T13:16:34.454358+0000","last_undegraded":"2024-03-30T13:16:34.454358+0000","last_fullsized":"2024-03-30T13:16:34.454358+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:16:09.352215+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:16:09.352215+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:32.702881+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1421,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-30T13:16:30.435963+0000","last_change":"2024-03-30T13:15:36.454725+0000","last_active":"2024-03-30T13:16:30.435963+0000","last_peered":"2024-03-30T13:16:30.435963+0000","last_clean":"2024-03-30T13:16:30.435963+0000","last_became_active":"2024-03-30T13:14:38.238174+0000","last_became_peered":"2024-03-30T13:14:38.238174+0000","last_unstale":"2024-03-30T13:16:30.435963+0000","last_undegraded":"2024-03-30T13:16:30.435963+0000","last_fullsized":"2024-03-30T13:16:30.435963+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:36.454575+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:15:36.454575+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:40.210170+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":71059072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1062,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-03-30T13:16:33.446423+0000","last_change":"2024-03-30T13:16:20.318505+0000","last_active":"2024-03-30T13:16:33.446423+0000","last_peered":"2024-03-30T13:16:33.446423+0000","last_clean":"2024-03-30T13:16:33.446423+0000","last_became_active":"2024-03-30T13:16:20.315944+0000","last_became_peered":"2024-03-30T13:16:20.315944+0000","last_unstale":"2024-03-30T13:16:33.446423+0000","last_undegraded":"2024-03-30T13:16:33.446423+0000","last_fullsized":"2024-03-30T13:16:33.446423+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_clean_scrub_stamp":"2024-03-30T13:16:16.623820+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:35.532032+0000","scrub_duration":659,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1326,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-30T13:16:30.609908+0000","last_change":"2024-03-30T13:16:20.314679+0000","last_active":"2024-03-30T13:16:30.609908+0000","last_peered":"2024-03-30T13:16:30.609908+0000","last_clean":"2024-03-30T13:16:30.609908+0000","last_became_active":"2024-03-30T13:16:20.313826+0000","last_became_peered":"2024-03-30T13:16:20.313826+0000","last_unstale":"2024-03-30T13:16:30.609908+0000","last_undegraded":"2024-03-30T13:16:30.609908+0000","last_fullsized":"2024-03-30T13:16:30.609908+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:15:27.030418+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:15:27.030418+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:31.173280+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":38218623,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1396,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-30T13:16:30.610149+0000","last_change":"2024-03-30T13:15:12.093216+0000","last_active":"2024-03-30T13:16:30.610149+0000","last_peered":"2024-03-30T13:16:30.610149+0000","last_clean":"2024-03-30T13:16:30.610149+0000","last_became_active":"2024-03-30T13:14:38.236424+0000","last_became_peered":"2024-03-30T13:14:38.236424+0000","last_unstale":"2024-03-30T13:16:30.610149+0000","last_undegraded":"2024-03-30T13:16:30.610149+0000","last_fullsized":"2024-03-30T13:16:30.610149+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:15:12.092921+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:34.616328+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":76975004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":160,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-30T13:16:30.424257+0000","last_change":"2024-03-30T13:15:14.320075+0000","last_active":"2024-03-30T13:16:30.424257+0000","last_peered":"2024-03-30T13:16:30.424257+0000","last_clean":"2024-03-30T13:16:30.424257+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:16:30.424257+0000","last_undegraded":"2024-03-30T13:16:30.424257+0000","last_fullsized":"2024-03-30T13:16:30.424257+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:15:14.319813+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:15:14.319813+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:34.069151+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":756,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-30T13:16:32.447050+0000","last_change":"2024-03-30T13:15:52.423713+0000","last_active":"2024-03-30T13:16:32.447050+0000","last_peered":"2024-03-30T13:16:32.447050+0000","last_clean":"2024-03-30T13:16:32.447050+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:16:32.447050+0000","last_undegraded":"2024-03-30T13:16:32.447050+0000","last_fullsized":"2024-03-30T13:16:32.447050+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:15:52.423297+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:15:52.423297+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:08.679223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":144,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-30T13:16:30.424306+0000","last_change":"2024-03-30T13:15:39.529282+0000","last_active":"2024-03-30T13:16:30.424306+0000","last_peered":"2024-03-30T13:16:30.424306+0000","last_clean":"2024-03-30T13:16:30.424306+0000","last_became_active":"2024-03-30T13:14:38.235512+0000","last_became_peered":"2024-03-30T13:14:38.235512+0000","last_unstale":"2024-03-30T13:16:30.424306+0000","last_undegraded":"2024-03-30T13:16:30.424306+0000","last_fullsized":"2024-03-30T13:16:30.424306+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:15:39.528997+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:15:39.528997+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:00.848645+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":534,"reported_epoch":574,"state":"active+clean","last_fresh":"2024-03-30T13:16:34.454380+0000","last_change":"2024-03-30T13:15:10.510582+0000","last_active":"2024-03-30T13:16:34.454380+0000","last_peered":"2024-03-30T13:16:34.454380+0000","last_clean":"2024-03-30T13:16:34.454380+0000","last_became_active":"2024-03-30T13:14:38.231065+0000","last_became_peered":"2024-03-30T13:14:38.231065+0000","last_unstale":"2024-03-30T13:16:34.454380+0000","last_undegraded":"2024-03-30T13:16:34.454380+0000","last_fullsized":"2024-03-30T13:16:34.454380+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:10.510312+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:15:10.510312+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:36.027524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15680733,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":446942545,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834199,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58204,"kb_used_data":21900,"kb_used_omap":5,"kb_used_meta":36282,"kb_avail":94313636,"statfs":{"total":96636764160,"available":96577163264,"internally_reserved":0,"allocated":22425600,"data_stored":18604307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":37152866},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157465,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64288,"kb_used_data":26644,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94307552,"statfs":{"total":96636764160,"available":96570933248,"internally_reserved":0,"allocated":27283456,"data_stored":23462163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":559,"seq":2400886718469,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56028,"kb_used_data":18384,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94315812,"statfs":{"total":96636764160,"available":96579391488,"internally_reserved":0,"allocated":18825216,"data_stored":14983520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12639,"internal_metadata":38522529},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509696,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48496,"kb_used_data":13152,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323344,"statfs":{"total":96636764160,"available":96587104256,"internally_reserved":0,"allocated":13467648,"data_stored":9625952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":461,"seq":1979979923503,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49188,"kb_used_data":12440,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94322652,"statfs":{"total":96636764160,"available":96586395648,"internally_reserved":0,"allocated":12738560,"data_stored":8890063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7692,"internal_metadata":37609972},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509697,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51516,"kb_used_data":20328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320324,"statfs":{"total":96636764160,"available":96584011776,"internally_reserved":0,"allocated":20815872,"data_stored":16974176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297091,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54508,"kb_used_data":16860,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94317332,"statfs":{"total":96636764160,"available":96580947968,"internally_reserved":0,"allocated":17264640,"data_stored":13429745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147405,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50988,"kb_used_data":19292,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94320852,"statfs":{"total":96636764160,"available":96584552448,"internally_reserved":0,"allocated":19755008,"data_stored":15933715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:16:36.160 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:16:36.160 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:16:36.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:36.271+0000 7fe9f6ee6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:36.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:36.283+0000 7fe9f6ee6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:36.456 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:16:36.456 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":575,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:16:35.458507+0000","last_up_change":"2024-03-30T13:16:19.279518+0000","last_in_change":"2024-03-30T13:16:19.279518+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":29,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":553,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":346,"last_clean_end":457,"up_from":461,"up_thru":515,"down_at":458,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2034332759},{"type":"v1","addr":"172.21.15.67:6818","nonce":2034332759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2034332759},{"type":"v1","addr":"172.21.15.67:6821","nonce":2034332759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2034332759},{"type":"v1","addr":"172.21.15.67:6828","nonce":2034332759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2034332759},{"type":"v1","addr":"172.21.15.67:6825","nonce":2034332759}]},"public_addr":"172.21.15.67:6818/2034332759","cluster_addr":"172.21.15.67:6821/2034332759","heartbeat_back_addr":"172.21.15.67:6828/2034332759","heartbeat_front_addr":"172.21.15.67:6825/2034332759","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":553,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":388,"up_from":393,"up_thru":515,"down_at":389,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6801","nonce":3837194718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6803","nonce":3837194718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6807","nonce":3837194718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6805","nonce":3837194718}]},"public_addr":"172.21.15.67:6801/3837194718","cluster_addr":"172.21.15.67:6803/3837194718","heartbeat_back_addr":"172.21.15.67:6807/3837194718","heartbeat_front_addr":"172.21.15.67:6805/3837194718","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":536,"last_clean_end":552,"up_from":559,"up_thru":559,"down_at":553,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3095958959},{"type":"v1","addr":"172.21.15.82:6803","nonce":3095958959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":3095958959},{"type":"v1","addr":"172.21.15.82:6807","nonce":3095958959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":3095958959},{"type":"v1","addr":"172.21.15.82:6815","nonce":3095958959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":3095958959},{"type":"v1","addr":"172.21.15.82:6811","nonce":3095958959}]},"public_addr":"172.21.15.82:6803/3095958959","cluster_addr":"172.21.15.82:6807/3095958959","heartbeat_back_addr":"172.21.15.82:6815/3095958959","heartbeat_front_addr":"172.21.15.82:6811/3095958959","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":488,"last_clean_end":506,"up_from":515,"up_thru":515,"down_at":507,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6817","nonce":292505453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6819","nonce":292505453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6823","nonce":292505453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6821","nonce":292505453}]},"public_addr":"172.21.15.82:6817/292505453","cluster_addr":"172.21.15.82:6819/292505453","heartbeat_back_addr":"172.21.15.82:6823/292505453","heartbeat_front_addr":"172.21.15.82:6821/292505453","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":227,"last_clean_end":363,"up_from":372,"up_thru":559,"down_at":364,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6801","nonce":189641108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6806","nonce":189641108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6814","nonce":189641108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6810","nonce":189641108}]},"public_addr":"172.21.15.82:6801/189641108","cluster_addr":"172.21.15.82:6806/189641108","heartbeat_back_addr":"172.21.15.82:6814/189641108","heartbeat_front_addr":"172.21.15.82:6810/189641108","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":298,"last_clean_end":409,"up_from":419,"up_thru":559,"down_at":410,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6825","nonce":2166599911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6827","nonce":2166599911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6831","nonce":2166599911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6829","nonce":2166599911}]},"public_addr":"172.21.15.82:6825/2166599911","cluster_addr":"172.21.15.82:6827/2166599911","heartbeat_back_addr":"172.21.15.82:6831/2166599911","heartbeat_front_addr":"172.21.15.82:6829/2166599911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:16:10.169160+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:16:29.677248+0000","dead_epoch":458},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:15:28.906751+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:16:01.634626+0000","dead_epoch":389},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:15:36.571402+0000","dead_epoch":553},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:15:50.735114+0000","dead_epoch":507},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:15:39.514759+0000","dead_epoch":364},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:15:40.057282+0000","dead_epoch":410}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:16:36.474 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:16:36.475 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:16:36.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:36.587+0000 7fdfc4030640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:36.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:36.603+0000 7fdfc4030640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:36.780 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:16:36.799 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:16:36.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:36.911+0000 7f27b934f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:36.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:36.927+0000 7f27b934f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:37.103 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:16:37.103 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:16:37.120 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:16:37.120 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:16:37.120 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:16:37.120 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:16:37.121 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 6, 3, 7, 1, 5, 4] 2024-03-30T13:16:37.121 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:16:37.121 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:16:37.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:37.223+0000 7f2b3ce15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:37.235+0000 7f2b3ce15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:37.412 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:16:37.413 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:16:37.431 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1200,"stamp":"2024-03-30T13:16:36.148098+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":446942545,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":433216,"kb_used_data":149000,"kb_used_omap":56,"kb_used_meta":284039,"kb_avail":754541504,"statfs":{"total":773094113280,"available":772650500096,"internally_reserved":0,"allocated":152576000,"data_stored":121903641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":57603,"internal_metadata":290856701},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001114"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1342,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-30T13:16:30.610334+0000","last_change":"2024-03-30T13:16:20.317770+0000","last_active":"2024-03-30T13:16:30.610334+0000","last_peered":"2024-03-30T13:16:30.610334+0000","last_clean":"2024-03-30T13:16:30.610334+0000","last_became_active":"2024-03-30T13:16:20.316817+0000","last_became_peered":"2024-03-30T13:16:20.316817+0000","last_unstale":"2024-03-30T13:16:30.610334+0000","last_undegraded":"2024-03-30T13:16:30.610334+0000","last_fullsized":"2024-03-30T13:16:30.610334+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:31.536364+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:15:31.536364+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:48.846664+0000","scrub_duration":515,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":32117715,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1190,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-03-30T13:16:33.446532+0000","last_change":"2024-03-30T13:16:20.315116+0000","last_active":"2024-03-30T13:16:33.446532+0000","last_peered":"2024-03-30T13:16:33.446532+0000","last_clean":"2024-03-30T13:16:33.446532+0000","last_became_active":"2024-03-30T13:16:20.313939+0000","last_became_peered":"2024-03-30T13:16:20.313939+0000","last_unstale":"2024-03-30T13:16:33.446532+0000","last_undegraded":"2024-03-30T13:16:33.446532+0000","last_fullsized":"2024-03-30T13:16:33.446532+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:16:02.397699+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:16:02.397699+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:29.215054+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":884,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-30T13:16:32.446883+0000","last_change":"2024-03-30T13:16:20.312577+0000","last_active":"2024-03-30T13:16:32.446883+0000","last_peered":"2024-03-30T13:16:32.446883+0000","last_clean":"2024-03-30T13:16:32.446883+0000","last_became_active":"2024-03-30T13:16:20.311258+0000","last_became_peered":"2024-03-30T13:16:20.311258+0000","last_unstale":"2024-03-30T13:16:32.446883+0000","last_undegraded":"2024-03-30T13:16:32.446883+0000","last_fullsized":"2024-03-30T13:16:32.446883+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:16:04.484050+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:16:04.484050+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:18.836317+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":46890936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":785,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-30T13:16:32.446828+0000","last_change":"2024-03-30T13:16:20.586918+0000","last_active":"2024-03-30T13:16:32.446828+0000","last_peered":"2024-03-30T13:16:32.446828+0000","last_clean":"2024-03-30T13:16:32.446828+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:16:32.446828+0000","last_undegraded":"2024-03-30T13:16:32.446828+0000","last_fullsized":"2024-03-30T13:16:32.446828+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:20.586668+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:16:20.586668+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:50.224020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":147,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-30T13:16:30.424286+0000","last_change":"2024-03-30T13:15:21.424552+0000","last_active":"2024-03-30T13:16:30.424286+0000","last_peered":"2024-03-30T13:16:30.424286+0000","last_clean":"2024-03-30T13:16:30.424286+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:16:30.424286+0000","last_undegraded":"2024-03-30T13:16:30.424286+0000","last_fullsized":"2024-03-30T13:16:30.424286+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:15:21.424313+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:15:21.424313+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:49.667982+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1320,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-30T13:16:30.435851+0000","last_change":"2024-03-30T13:16:19.587276+0000","last_active":"2024-03-30T13:16:30.435851+0000","last_peered":"2024-03-30T13:16:30.435851+0000","last_clean":"2024-03-30T13:16:30.435851+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:16:30.435851+0000","last_undegraded":"2024-03-30T13:16:30.435851+0000","last_fullsized":"2024-03-30T13:16:30.435851+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:16:19.587123+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:16:19.587123+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:25.576107+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":872,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-30T13:16:32.446858+0000","last_change":"2024-03-30T13:16:20.314213+0000","last_active":"2024-03-30T13:16:32.446858+0000","last_peered":"2024-03-30T13:16:32.446858+0000","last_clean":"2024-03-30T13:16:32.446858+0000","last_became_active":"2024-03-30T13:16:20.312300+0000","last_became_peered":"2024-03-30T13:16:20.312300+0000","last_unstale":"2024-03-30T13:16:32.446858+0000","last_undegraded":"2024-03-30T13:16:32.446858+0000","last_fullsized":"2024-03-30T13:16:32.446858+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:15:31.305358+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:15:31.305358+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:57.539681+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58554796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":591,"reported_epoch":574,"state":"active+clean","last_fresh":"2024-03-30T13:16:34.454358+0000","last_change":"2024-03-30T13:16:09.352505+0000","last_active":"2024-03-30T13:16:34.454358+0000","last_peered":"2024-03-30T13:16:34.454358+0000","last_clean":"2024-03-30T13:16:34.454358+0000","last_became_active":"2024-03-30T13:14:38.233325+0000","last_became_peered":"2024-03-30T13:14:38.233325+0000","last_unstale":"2024-03-30T13:16:34.454358+0000","last_undegraded":"2024-03-30T13:16:34.454358+0000","last_fullsized":"2024-03-30T13:16:34.454358+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:16:09.352215+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:16:09.352215+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:32.702881+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1421,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-30T13:16:30.435963+0000","last_change":"2024-03-30T13:15:36.454725+0000","last_active":"2024-03-30T13:16:30.435963+0000","last_peered":"2024-03-30T13:16:30.435963+0000","last_clean":"2024-03-30T13:16:30.435963+0000","last_became_active":"2024-03-30T13:14:38.238174+0000","last_became_peered":"2024-03-30T13:14:38.238174+0000","last_unstale":"2024-03-30T13:16:30.435963+0000","last_undegraded":"2024-03-30T13:16:30.435963+0000","last_fullsized":"2024-03-30T13:16:30.435963+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:36.454575+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:15:36.454575+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:40.210170+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":71059072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1062,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-03-30T13:16:33.446423+0000","last_change":"2024-03-30T13:16:20.318505+0000","last_active":"2024-03-30T13:16:33.446423+0000","last_peered":"2024-03-30T13:16:33.446423+0000","last_clean":"2024-03-30T13:16:33.446423+0000","last_became_active":"2024-03-30T13:16:20.315944+0000","last_became_peered":"2024-03-30T13:16:20.315944+0000","last_unstale":"2024-03-30T13:16:33.446423+0000","last_undegraded":"2024-03-30T13:16:33.446423+0000","last_fullsized":"2024-03-30T13:16:33.446423+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_clean_scrub_stamp":"2024-03-30T13:16:16.623820+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:35.532032+0000","scrub_duration":659,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1326,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-30T13:16:30.609908+0000","last_change":"2024-03-30T13:16:20.314679+0000","last_active":"2024-03-30T13:16:30.609908+0000","last_peered":"2024-03-30T13:16:30.609908+0000","last_clean":"2024-03-30T13:16:30.609908+0000","last_became_active":"2024-03-30T13:16:20.313826+0000","last_became_peered":"2024-03-30T13:16:20.313826+0000","last_unstale":"2024-03-30T13:16:30.609908+0000","last_undegraded":"2024-03-30T13:16:30.609908+0000","last_fullsized":"2024-03-30T13:16:30.609908+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:15:27.030418+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:15:27.030418+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:31.173280+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":38218623,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1396,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-30T13:16:30.610149+0000","last_change":"2024-03-30T13:15:12.093216+0000","last_active":"2024-03-30T13:16:30.610149+0000","last_peered":"2024-03-30T13:16:30.610149+0000","last_clean":"2024-03-30T13:16:30.610149+0000","last_became_active":"2024-03-30T13:14:38.236424+0000","last_became_peered":"2024-03-30T13:14:38.236424+0000","last_unstale":"2024-03-30T13:16:30.610149+0000","last_undegraded":"2024-03-30T13:16:30.610149+0000","last_fullsized":"2024-03-30T13:16:30.610149+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:15:12.092921+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:34.616328+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":76975004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":160,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-30T13:16:30.424257+0000","last_change":"2024-03-30T13:15:14.320075+0000","last_active":"2024-03-30T13:16:30.424257+0000","last_peered":"2024-03-30T13:16:30.424257+0000","last_clean":"2024-03-30T13:16:30.424257+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:16:30.424257+0000","last_undegraded":"2024-03-30T13:16:30.424257+0000","last_fullsized":"2024-03-30T13:16:30.424257+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:15:14.319813+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:15:14.319813+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:34.069151+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":756,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-03-30T13:16:32.447050+0000","last_change":"2024-03-30T13:15:52.423713+0000","last_active":"2024-03-30T13:16:32.447050+0000","last_peered":"2024-03-30T13:16:32.447050+0000","last_clean":"2024-03-30T13:16:32.447050+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:16:32.447050+0000","last_undegraded":"2024-03-30T13:16:32.447050+0000","last_fullsized":"2024-03-30T13:16:32.447050+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:15:52.423297+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:15:52.423297+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:08.679223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":144,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-03-30T13:16:30.424306+0000","last_change":"2024-03-30T13:15:39.529282+0000","last_active":"2024-03-30T13:16:30.424306+0000","last_peered":"2024-03-30T13:16:30.424306+0000","last_clean":"2024-03-30T13:16:30.424306+0000","last_became_active":"2024-03-30T13:14:38.235512+0000","last_became_peered":"2024-03-30T13:14:38.235512+0000","last_unstale":"2024-03-30T13:16:30.424306+0000","last_undegraded":"2024-03-30T13:16:30.424306+0000","last_fullsized":"2024-03-30T13:16:30.424306+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:15:39.528997+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:15:39.528997+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:00.848645+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":534,"reported_epoch":574,"state":"active+clean","last_fresh":"2024-03-30T13:16:34.454380+0000","last_change":"2024-03-30T13:15:10.510582+0000","last_active":"2024-03-30T13:16:34.454380+0000","last_peered":"2024-03-30T13:16:34.454380+0000","last_clean":"2024-03-30T13:16:34.454380+0000","last_became_active":"2024-03-30T13:14:38.231065+0000","last_became_peered":"2024-03-30T13:14:38.231065+0000","last_unstale":"2024-03-30T13:16:34.454380+0000","last_undegraded":"2024-03-30T13:16:34.454380+0000","last_fullsized":"2024-03-30T13:16:34.454380+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:10.510312+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:15:10.510312+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:36.027524+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15680733,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":446942545,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834199,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58204,"kb_used_data":21900,"kb_used_omap":5,"kb_used_meta":36282,"kb_avail":94313636,"statfs":{"total":96636764160,"available":96577163264,"internally_reserved":0,"allocated":22425600,"data_stored":18604307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":37152866},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157465,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64288,"kb_used_data":26644,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94307552,"statfs":{"total":96636764160,"available":96570933248,"internally_reserved":0,"allocated":27283456,"data_stored":23462163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":559,"seq":2400886718469,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56028,"kb_used_data":18384,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94315812,"statfs":{"total":96636764160,"available":96579391488,"internally_reserved":0,"allocated":18825216,"data_stored":14983520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12639,"internal_metadata":38522529},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509696,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48496,"kb_used_data":13152,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323344,"statfs":{"total":96636764160,"available":96587104256,"internally_reserved":0,"allocated":13467648,"data_stored":9625952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":461,"seq":1979979923503,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49188,"kb_used_data":12440,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94322652,"statfs":{"total":96636764160,"available":96586395648,"internally_reserved":0,"allocated":12738560,"data_stored":8890063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7692,"internal_metadata":37609972},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509697,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51516,"kb_used_data":20328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320324,"statfs":{"total":96636764160,"available":96584011776,"internally_reserved":0,"allocated":20815872,"data_stored":16974176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297091,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54508,"kb_used_data":16860,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94317332,"statfs":{"total":96636764160,"available":96580947968,"internally_reserved":0,"allocated":17264640,"data_stored":13429745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147405,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50988,"kb_used_data":19292,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94320852,"statfs":{"total":96636764160,"available":96584552448,"internally_reserved":0,"allocated":19755008,"data_stored":15933715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:16:37.432 INFO:tasks.thrashosds.thrasher:Choosing PG 1.3 with acting set [1, 6, 5] 2024-03-30T13:16:37.432 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 2, 6, 3, 7, 1, 5, 4] 2024-03-30T13:16:37.432 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-30T13:16:37.432 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:16:40.417 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-03-30T13:16:43.535 INFO:tasks.ceph.osd.1:Stopped 2024-03-30T13:16:43.535 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 2, 6, 3, 7, 1, 5, 4] 2024-03-30T13:16:43.535 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 1 2024-03-30T13:16:43.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:43.663+0000 7f420790a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:43.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:43.675+0000 7f420790a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:43.861 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:16:43.859+0000 7fbb38f08640 -1 mon.a@0(leader).osd e577 definitely_dead 0 2024-03-30T13:16:43.922 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.1. 2024-03-30T13:16:43.945 INFO:tasks.thrashosds.thrasher:dead_osds=[1], live_osds=[0, 2, 6, 3, 7, 5, 4] 2024-03-30T13:16:43.945 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:16:43.946 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:16:44.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:44.059+0000 7f9f7a23b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:44.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:44.071+0000 7f9f7a23b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:44.242 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:16:44.242 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:16:44.259 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1207,"stamp":"2024-03-30T13:16:44.150388+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":365160772,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":45,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":384448,"kb_used_data":136980,"kb_used_omap":48,"kb_used_meta":247311,"kb_avail":660218432,"statfs":{"total":676457349120,"available":676063674368,"internally_reserved":0,"allocated":140267520,"data_stored":113206381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":49911,"internal_metadata":253246729},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-33,"num_bytes_recovered":-81781773,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001793"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1356,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-30T13:16:39.520826+0000","last_change":"2024-03-30T13:16:20.317770+0000","last_active":"2024-03-30T13:16:39.520826+0000","last_peered":"2024-03-30T13:16:39.520826+0000","last_clean":"2024-03-30T13:16:39.520826+0000","last_became_active":"2024-03-30T13:16:20.316817+0000","last_became_peered":"2024-03-30T13:16:20.316817+0000","last_unstale":"2024-03-30T13:16:39.520826+0000","last_undegraded":"2024-03-30T13:16:39.520826+0000","last_fullsized":"2024-03-30T13:16:39.520826+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:31.536364+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:15:31.536364+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:48.846664+0000","scrub_duration":515,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":32117715,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1196,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-03-30T13:16:38.505655+0000","last_change":"2024-03-30T13:16:20.315116+0000","last_active":"2024-03-30T13:16:38.505655+0000","last_peered":"2024-03-30T13:16:38.505655+0000","last_clean":"2024-03-30T13:16:38.505655+0000","last_became_active":"2024-03-30T13:16:20.313939+0000","last_became_peered":"2024-03-30T13:16:20.313939+0000","last_unstale":"2024-03-30T13:16:38.505655+0000","last_undegraded":"2024-03-30T13:16:38.505655+0000","last_fullsized":"2024-03-30T13:16:38.505655+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:16:02.397699+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:16:02.397699+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:29.215054+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":897,"reported_epoch":577,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:16:39.526356+0000","last_change":"2024-03-30T13:16:39.526134+0000","last_active":"2024-03-30T13:16:39.526356+0000","last_peered":"2024-03-30T13:16:39.526356+0000","last_clean":"2024-03-30T13:16:36.085467+0000","last_became_active":"2024-03-30T13:16:39.526134+0000","last_became_peered":"2024-03-30T13:16:39.526134+0000","last_unstale":"2024-03-30T13:16:39.526356+0000","last_undegraded":"2024-03-30T13:16:39.520687+0000","last_fullsized":"2024-03-30T13:16:39.520353+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:16:04.484050+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:16:04.484050+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":46890936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2147483647],"acting":[7,4,2147483647],"avail_no_missing":["7(0)","4(1)"],"object_location_counts":[{"shards":"4(1),7(0)","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":795,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-30T13:16:39.520071+0000","last_change":"2024-03-30T13:16:20.586918+0000","last_active":"2024-03-30T13:16:39.520071+0000","last_peered":"2024-03-30T13:16:39.520071+0000","last_clean":"2024-03-30T13:16:39.520071+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:16:39.520071+0000","last_undegraded":"2024-03-30T13:16:39.520071+0000","last_fullsized":"2024-03-30T13:16:39.520071+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:20.586668+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:16:20.586668+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:50.224020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":161,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-30T13:16:39.516938+0000","last_change":"2024-03-30T13:15:21.424552+0000","last_active":"2024-03-30T13:16:39.516938+0000","last_peered":"2024-03-30T13:16:39.516938+0000","last_clean":"2024-03-30T13:16:39.516938+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:16:39.516938+0000","last_undegraded":"2024-03-30T13:16:39.516938+0000","last_fullsized":"2024-03-30T13:16:39.516938+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:15:21.424313+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:15:21.424313+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:49.667982+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1333,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-30T13:16:40.345240+0000","last_change":"2024-03-30T13:16:19.587276+0000","last_active":"2024-03-30T13:16:40.345240+0000","last_peered":"2024-03-30T13:16:40.345240+0000","last_clean":"2024-03-30T13:16:40.345240+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:16:40.345240+0000","last_undegraded":"2024-03-30T13:16:40.345240+0000","last_fullsized":"2024-03-30T13:16:40.345240+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:16:19.587123+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:16:19.587123+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:25.576107+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":882,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-30T13:16:39.520556+0000","last_change":"2024-03-30T13:16:20.314213+0000","last_active":"2024-03-30T13:16:39.520556+0000","last_peered":"2024-03-30T13:16:39.520556+0000","last_clean":"2024-03-30T13:16:39.520556+0000","last_became_active":"2024-03-30T13:16:20.312300+0000","last_became_peered":"2024-03-30T13:16:20.312300+0000","last_unstale":"2024-03-30T13:16:39.520556+0000","last_undegraded":"2024-03-30T13:16:39.520556+0000","last_fullsized":"2024-03-30T13:16:39.520556+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:15:31.305358+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:15:31.305358+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:16:57.539681+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58554796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":35,"reported_epoch":577,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:16:39.525160+0000","last_change":"2024-03-30T13:16:39.524219+0000","last_active":"2024-03-30T13:16:39.525160+0000","last_peered":"2024-03-30T13:16:39.525160+0000","last_clean":"2024-03-30T12:55:47.556542+0000","last_became_active":"2024-03-30T13:16:39.524219+0000","last_became_peered":"2024-03-30T13:16:39.524219+0000","last_unstale":"2024-03-30T13:16:39.525160+0000","last_undegraded":"2024-03-30T13:16:39.520514+0000","last_fullsized":"2024-03-30T13:16:39.520156+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:16:09.352215+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:16:09.352215+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:14.551664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,6,5],"acting":[2147483647,6,5],"avail_no_missing":["6(1)","5(2)"],"object_location_counts":[{"shards":"5(2),6(1)","objects":3}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1434,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-30T13:16:40.345234+0000","last_change":"2024-03-30T13:15:36.454725+0000","last_active":"2024-03-30T13:16:40.345234+0000","last_peered":"2024-03-30T13:16:40.345234+0000","last_clean":"2024-03-30T13:16:40.345234+0000","last_became_active":"2024-03-30T13:14:38.238174+0000","last_became_peered":"2024-03-30T13:14:38.238174+0000","last_unstale":"2024-03-30T13:16:40.345234+0000","last_undegraded":"2024-03-30T13:16:40.345234+0000","last_fullsized":"2024-03-30T13:16:40.345234+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:36.454575+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:10:52.360707+0000","last_clean_scrub_stamp":"2024-03-30T13:15:36.454575+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":71059072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1068,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-03-30T13:16:38.505573+0000","last_change":"2024-03-30T13:16:20.318505+0000","last_active":"2024-03-30T13:16:38.505573+0000","last_peered":"2024-03-30T13:16:38.505573+0000","last_clean":"2024-03-30T13:16:38.505573+0000","last_became_active":"2024-03-30T13:16:20.315944+0000","last_became_peered":"2024-03-30T13:16:20.315944+0000","last_unstale":"2024-03-30T13:16:38.505573+0000","last_undegraded":"2024-03-30T13:16:38.505573+0000","last_fullsized":"2024-03-30T13:16:38.505573+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_clean_scrub_stamp":"2024-03-30T13:16:16.623820+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:35.532032+0000","scrub_duration":659,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1348,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-30T13:16:39.520577+0000","last_change":"2024-03-30T13:16:32.758023+0000","last_active":"2024-03-30T13:16:39.520577+0000","last_peered":"2024-03-30T13:16:39.520577+0000","last_clean":"2024-03-30T13:16:39.520577+0000","last_became_active":"2024-03-30T13:16:20.313826+0000","last_became_peered":"2024-03-30T13:16:20.313826+0000","last_unstale":"2024-03-30T13:16:39.520577+0000","last_undegraded":"2024-03-30T13:16:39.520577+0000","last_fullsized":"2024-03-30T13:16:39.520577+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:16:32.757775+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:16:32.757775+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:50.377606+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":38218623,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1418,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-30T13:16:39.520748+0000","last_change":"2024-03-30T13:16:35.688621+0000","last_active":"2024-03-30T13:16:39.520748+0000","last_peered":"2024-03-30T13:16:39.520748+0000","last_clean":"2024-03-30T13:16:39.520748+0000","last_became_active":"2024-03-30T13:14:38.236424+0000","last_became_peered":"2024-03-30T13:14:38.236424+0000","last_unstale":"2024-03-30T13:16:39.520748+0000","last_undegraded":"2024-03-30T13:16:39.520748+0000","last_fullsized":"2024-03-30T13:16:39.520748+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:16:35.688353+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:16:35.688353+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:37.650028+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":76975004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":181,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-30T13:16:39.516920+0000","last_change":"2024-03-30T13:16:37.690115+0000","last_active":"2024-03-30T13:16:39.516920+0000","last_peered":"2024-03-30T13:16:39.516920+0000","last_clean":"2024-03-30T13:16:39.516920+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:16:39.516920+0000","last_undegraded":"2024-03-30T13:16:39.516920+0000","last_fullsized":"2024-03-30T13:16:39.516920+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:16:37.689862+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:16:37.689862+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:39.492459+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":766,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-30T13:16:39.520227+0000","last_change":"2024-03-30T13:15:52.423713+0000","last_active":"2024-03-30T13:16:39.520227+0000","last_peered":"2024-03-30T13:16:39.520227+0000","last_clean":"2024-03-30T13:16:39.520227+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:16:39.520227+0000","last_undegraded":"2024-03-30T13:16:39.520227+0000","last_fullsized":"2024-03-30T13:16:39.520227+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:15:52.423297+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:15:52.423297+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:08.679223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":157,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-03-30T13:16:39.516938+0000","last_change":"2024-03-30T13:15:39.529282+0000","last_active":"2024-03-30T13:16:39.516938+0000","last_peered":"2024-03-30T13:16:39.516938+0000","last_clean":"2024-03-30T13:16:39.516938+0000","last_became_active":"2024-03-30T13:14:38.235512+0000","last_became_peered":"2024-03-30T13:14:38.235512+0000","last_unstale":"2024-03-30T13:16:39.516938+0000","last_undegraded":"2024-03-30T13:16:39.516938+0000","last_fullsized":"2024-03-30T13:16:39.516938+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:15:39.528997+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:15:39.528997+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:00.848645+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":31,"reported_epoch":577,"state":"active+undersized+degraded","last_fresh":"2024-03-30T13:16:39.516741+0000","last_change":"2024-03-30T13:16:39.516207+0000","last_active":"2024-03-30T13:16:39.516741+0000","last_peered":"2024-03-30T13:16:39.516741+0000","last_clean":"2024-03-30T12:55:47.600950+0000","last_became_active":"2024-03-30T13:16:39.516207+0000","last_became_peered":"2024-03-30T13:16:39.516207+0000","last_unstale":"2024-03-30T13:16:39.516741+0000","last_undegraded":"2024-03-30T13:16:39.513001+0000","last_fullsized":"2024-03-30T13:16:39.512790+0000","mapping_epoch":576,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:15:10.510312+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:15:10.510312+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,0,5],"acting":[2147483647,0,5],"avail_no_missing":["0(1)","5(2)"],"object_location_counts":[{"shards":"0(1),5(2)","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":365160772,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834201,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58288,"kb_used_data":21984,"kb_used_omap":5,"kb_used_meta":36282,"kb_avail":94313552,"statfs":{"total":96636764160,"available":96577077248,"internally_reserved":0,"allocated":22511616,"data_stored":18642845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":37152866},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157467,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64372,"kb_used_data":26728,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94307468,"statfs":{"total":96636764160,"available":96570847232,"internally_reserved":0,"allocated":27369472,"data_stored":23500701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":559,"seq":2400886718470,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56060,"kb_used_data":18408,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94315780,"statfs":{"total":96636764160,"available":96579358720,"internally_reserved":0,"allocated":18849792,"data_stored":14994494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12639,"internal_metadata":38522529},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509697,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48528,"kb_used_data":13188,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323312,"statfs":{"total":96636764160,"available":96587071488,"internally_reserved":0,"allocated":13504512,"data_stored":9642526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509698,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51560,"kb_used_data":20376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320280,"statfs":{"total":96636764160,"available":96583966720,"internally_reserved":0,"allocated":20865024,"data_stored":16996253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297093,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54568,"kb_used_data":16920,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94317272,"statfs":{"total":96636764160,"available":96580886528,"internally_reserved":0,"allocated":17326080,"data_stored":13457309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147407,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51072,"kb_used_data":19376,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94320768,"statfs":{"total":96636764160,"available":96584466432,"internally_reserved":0,"allocated":19841024,"data_stored":15972253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:16:44.260 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:16:44.261 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:16:44.261 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-03-30T13:16:44.261 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-30T13:16:44.261 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-30T13:16:44.263 INFO:tasks.ceph.osd.1:Started 2024-03-30T13:16:44.263 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-30T13:16:44.320 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:16:44.315+0000 7f2c2d6427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:44.327 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:16:44.323+0000 7f2c2d6427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:44.327 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:16:44.323+0000 7f2c2d6427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:44.370 INFO:teuthology.orchestra.run.smithi067.stderr:no valid command found; 10 closest matches: 2024-03-30T13:16:44.370 INFO:teuthology.orchestra.run.smithi067.stderr:0 2024-03-30T13:16:44.370 INFO:teuthology.orchestra.run.smithi067.stderr:1 2024-03-30T13:16:44.370 INFO:teuthology.orchestra.run.smithi067.stderr:2 2024-03-30T13:16:44.370 INFO:teuthology.orchestra.run.smithi067.stderr:abort 2024-03-30T13:16:44.370 INFO:teuthology.orchestra.run.smithi067.stderr:assert 2024-03-30T13:16:44.370 INFO:teuthology.orchestra.run.smithi067.stderr:config diff 2024-03-30T13:16:44.370 INFO:teuthology.orchestra.run.smithi067.stderr:config diff get 2024-03-30T13:16:44.370 INFO:teuthology.orchestra.run.smithi067.stderr:config get 2024-03-30T13:16:44.370 INFO:teuthology.orchestra.run.smithi067.stderr:config help [] 2024-03-30T13:16:44.371 INFO:teuthology.orchestra.run.smithi067.stderr:config set ... 2024-03-30T13:16:44.371 INFO:teuthology.orchestra.run.smithi067.stderr:admin_socket: invalid command 2024-03-30T13:16:44.372 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:16:44.373 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-30T13:16:44.875 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:16:44.871+0000 7f2c2d6427c0 -1 Falling back to public interface 2024-03-30T13:16:46.727 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-03-30T13:16:48.707 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:16:48.703+0000 7f2c2d6427c0 -1 osd.1 575 log_to_monitors true 2024-03-30T13:16:48.994 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:16:48.991+0000 7f2c2140f640 -1 osd.1 575 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:16:49.374 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-30T13:16:49.509 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-30T13:16:49.510 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-30T13:16:49.510 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-03-30T13:16:49.510 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-30T13:16:49.524 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:16:49.524 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-03-30T13:16:49.524 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 1 2024-03-30T13:16:49.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:49.639+0000 7ff0751b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:49.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:49.655+0000 7ff0751b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:49.838 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:16:49.835+0000 7fbb38f08640 -1 mon.a@0(leader).osd e580 definitely_dead 0 2024-03-30T13:16:49.957 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.1. 2024-03-30T13:16:49.988 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-03-30T13:16:49.988 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:16:50.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:50.103+0000 7fe0edbb0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:50.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:50.123+0000 7fe0edbb0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:50.978 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:16:51.006 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:16:51.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:51.119+0000 7f4cb3eff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:51.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:51.135+0000 7f4cb3eff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:51.989 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:16:52.015 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:16:52.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:52.131+0000 7fc8b59c1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:52.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:52.147+0000 7fc8b59c1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:53.001 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:16:53.032 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:16:53.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:53.151+0000 7f845753b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:53.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:53.167+0000 7f845753b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:54.010 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:16:54.034 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:16:54.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:54.155+0000 7fb695636640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:54.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:54.171+0000 7fb695636640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:55.014 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:16:55.042 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:16:55.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:55.159+0000 7f9b32fd4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:55.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:55.175+0000 7f9b32fd4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:56.035 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:16:56.060 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:16:56.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:56.175+0000 7fd3c0aef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:56.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:56.191+0000 7fd3c0aef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:57.057 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:16:57.082 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:16:57.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:57.199+0000 7ff10231b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:57.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:57.215+0000 7ff10231b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:58.061 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:16:58.089 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:16:58.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:58.203+0000 7fa43a4fe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:58.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:58.223+0000 7fa43a4fe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:59.076 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:16:59.100 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:16:59.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:59.219+0000 7f5e98353640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:16:59.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:16:59.235+0000 7f5e98353640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:00.086 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:17:00.117 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:17:00.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:00.235+0000 7fe8efa0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:00.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:00.251+0000 7fe8efa0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:01.096 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:17:01.123 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:17:01.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:01.239+0000 7fc5729de640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:01.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:01.255+0000 7fc5729de640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:02.103 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:17:02.129 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:17:02.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:02.247+0000 7fa9195c9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:02.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:02.263+0000 7fa9195c9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:03.114 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:17:03.144 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:17:03.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:03.263+0000 7fac33ae9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:03.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:03.279+0000 7fac33ae9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:04.116 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:17:04.141 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:17:04.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:04.259+0000 7fa4fe114640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:04.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:04.275+0000 7fa4fe114640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:05.128 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:17:05.158 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:17:05.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:05.275+0000 7f5cb26a1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:05.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:05.291+0000 7f5cb26a1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:06.142 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:17:06.170 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:17:06.170 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:17:06.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:06.287+0000 7fa2e8fed640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:06.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:06.299+0000 7fa2e8fed640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:06.476 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:17:06.477 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:17:06.494 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1237,"stamp":"2024-03-30T13:17:06.156452+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":431732408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":435700,"kb_used_data":151028,"kb_used_omap":57,"kb_used_meta":284486,"kb_avail":754539020,"statfs":{"total":773094113280,"available":772647956480,"internally_reserved":0,"allocated":154652672,"data_stored":122846008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59250,"internal_metadata":291313806},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001015"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1392,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-03-30T13:17:00.605130+0000","last_change":"2024-03-30T13:16:50.692000+0000","last_active":"2024-03-30T13:17:00.605130+0000","last_peered":"2024-03-30T13:17:00.605130+0000","last_clean":"2024-03-30T13:17:00.605130+0000","last_became_active":"2024-03-30T13:16:20.316817+0000","last_became_peered":"2024-03-30T13:16:20.316817+0000","last_unstale":"2024-03-30T13:17:00.605130+0000","last_undegraded":"2024-03-30T13:17:00.605130+0000","last_fullsized":"2024-03-30T13:17:00.605130+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:50.691736+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:16:50.691736+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:51.654673+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":32117715,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1234,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-30T13:17:04.218942+0000","last_change":"2024-03-30T13:16:20.315116+0000","last_active":"2024-03-30T13:17:04.218942+0000","last_peered":"2024-03-30T13:17:04.218942+0000","last_clean":"2024-03-30T13:17:04.218942+0000","last_became_active":"2024-03-30T13:16:20.313939+0000","last_became_peered":"2024-03-30T13:16:20.313939+0000","last_unstale":"2024-03-30T13:17:04.218942+0000","last_undegraded":"2024-03-30T13:17:04.218942+0000","last_fullsized":"2024-03-30T13:17:04.218942+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:16:02.397699+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:16:02.397699+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:29.215054+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":949,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-30T13:17:02.389627+0000","last_change":"2024-03-30T13:16:50.994562+0000","last_active":"2024-03-30T13:17:02.389627+0000","last_peered":"2024-03-30T13:17:02.389627+0000","last_clean":"2024-03-30T13:17:02.389627+0000","last_became_active":"2024-03-30T13:16:50.993492+0000","last_became_peered":"2024-03-30T13:16:50.993492+0000","last_unstale":"2024-03-30T13:17:02.389627+0000","last_undegraded":"2024-03-30T13:17:02.389627+0000","last_fullsized":"2024-03-30T13:17:02.389627+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:16:04.484050+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:16:04.484050+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:08.749359+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":54706092,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":827,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-30T13:17:02.389560+0000","last_change":"2024-03-30T13:16:20.586918+0000","last_active":"2024-03-30T13:17:02.389560+0000","last_peered":"2024-03-30T13:17:02.389560+0000","last_clean":"2024-03-30T13:17:02.389560+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:17:02.389560+0000","last_undegraded":"2024-03-30T13:17:02.389560+0000","last_fullsized":"2024-03-30T13:17:02.389560+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:20.586668+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:16:20.586668+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:50.224020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":197,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-03-30T13:17:00.095015+0000","last_change":"2024-03-30T13:16:50.770031+0000","last_active":"2024-03-30T13:17:00.095015+0000","last_peered":"2024-03-30T13:17:00.095015+0000","last_clean":"2024-03-30T13:17:00.095015+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:17:00.095015+0000","last_undegraded":"2024-03-30T13:17:00.095015+0000","last_fullsized":"2024-03-30T13:17:00.095015+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:16:50.769668+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:16:50.769668+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:04.116739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1360,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-03-30T13:17:00.601897+0000","last_change":"2024-03-30T13:16:19.587276+0000","last_active":"2024-03-30T13:17:00.601897+0000","last_peered":"2024-03-30T13:17:00.601897+0000","last_clean":"2024-03-30T13:17:00.601897+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:17:00.601897+0000","last_undegraded":"2024-03-30T13:17:00.601897+0000","last_fullsized":"2024-03-30T13:17:00.601897+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:16:19.587123+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:16:19.587123+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:25.576107+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":922,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-30T13:17:02.389619+0000","last_change":"2024-03-30T13:16:57.724504+0000","last_active":"2024-03-30T13:17:02.389619+0000","last_peered":"2024-03-30T13:17:02.389619+0000","last_clean":"2024-03-30T13:17:02.389619+0000","last_became_active":"2024-03-30T13:16:20.312300+0000","last_became_peered":"2024-03-30T13:16:20.312300+0000","last_unstale":"2024-03-30T13:17:02.389619+0000","last_undegraded":"2024-03-30T13:17:02.389619+0000","last_fullsized":"2024-03-30T13:17:02.389619+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:16:57.724219+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:16:57.724219+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:23.534886+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58554796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":592,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-30T13:17:04.120980+0000","last_change":"2024-03-30T13:16:51.187305+0000","last_active":"2024-03-30T13:17:04.120980+0000","last_peered":"2024-03-30T13:17:04.120980+0000","last_clean":"2024-03-30T13:17:04.120980+0000","last_became_active":"2024-03-30T13:16:51.186350+0000","last_became_peered":"2024-03-30T13:16:51.186350+0000","last_unstale":"2024-03-30T13:17:04.120980+0000","last_undegraded":"2024-03-30T13:17:04.120980+0000","last_fullsized":"2024-03-30T13:17:04.120980+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:16:09.352215+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:16:09.352215+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:32.403615+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1468,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-03-30T13:17:00.601534+0000","last_change":"2024-03-30T13:16:45.572619+0000","last_active":"2024-03-30T13:17:00.601534+0000","last_peered":"2024-03-30T13:17:00.601534+0000","last_clean":"2024-03-30T13:17:00.601534+0000","last_became_active":"2024-03-30T13:14:38.238174+0000","last_became_peered":"2024-03-30T13:14:38.238174+0000","last_unstale":"2024-03-30T13:17:00.601534+0000","last_undegraded":"2024-03-30T13:17:00.601534+0000","last_fullsized":"2024-03-30T13:17:00.601534+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:16:45.572362+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:06.255142+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":71059072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1106,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-30T13:17:04.218630+0000","last_change":"2024-03-30T13:16:20.318505+0000","last_active":"2024-03-30T13:17:04.218630+0000","last_peered":"2024-03-30T13:17:04.218630+0000","last_clean":"2024-03-30T13:17:04.218630+0000","last_became_active":"2024-03-30T13:16:20.315944+0000","last_became_peered":"2024-03-30T13:16:20.315944+0000","last_unstale":"2024-03-30T13:17:04.218630+0000","last_undegraded":"2024-03-30T13:17:04.218630+0000","last_fullsized":"2024-03-30T13:17:04.218630+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_clean_scrub_stamp":"2024-03-30T13:16:16.623820+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:35.532032+0000","scrub_duration":659,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1376,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-03-30T13:17:00.605330+0000","last_change":"2024-03-30T13:16:32.758023+0000","last_active":"2024-03-30T13:17:00.605330+0000","last_peered":"2024-03-30T13:17:00.605330+0000","last_clean":"2024-03-30T13:17:00.605330+0000","last_became_active":"2024-03-30T13:16:20.313826+0000","last_became_peered":"2024-03-30T13:16:20.313826+0000","last_unstale":"2024-03-30T13:17:00.605330+0000","last_undegraded":"2024-03-30T13:17:00.605330+0000","last_fullsized":"2024-03-30T13:17:00.605330+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:16:32.757775+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:16:32.757775+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:50.377606+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":38218623,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1446,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-03-30T13:17:00.605453+0000","last_change":"2024-03-30T13:16:35.688621+0000","last_active":"2024-03-30T13:17:00.605453+0000","last_peered":"2024-03-30T13:17:00.605453+0000","last_clean":"2024-03-30T13:17:00.605453+0000","last_became_active":"2024-03-30T13:14:38.236424+0000","last_became_peered":"2024-03-30T13:14:38.236424+0000","last_unstale":"2024-03-30T13:17:00.605453+0000","last_undegraded":"2024-03-30T13:17:00.605453+0000","last_fullsized":"2024-03-30T13:17:00.605453+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:16:35.688353+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:16:35.688353+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:37.650028+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":76975004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":210,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-03-30T13:17:00.094961+0000","last_change":"2024-03-30T13:16:37.690115+0000","last_active":"2024-03-30T13:17:00.094961+0000","last_peered":"2024-03-30T13:17:00.094961+0000","last_clean":"2024-03-30T13:17:00.094961+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:17:00.094961+0000","last_undegraded":"2024-03-30T13:17:00.094961+0000","last_fullsized":"2024-03-30T13:17:00.094961+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:16:37.689862+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:16:37.689862+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:39.492459+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":798,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-30T13:17:02.389568+0000","last_change":"2024-03-30T13:15:52.423713+0000","last_active":"2024-03-30T13:17:02.389568+0000","last_peered":"2024-03-30T13:17:02.389568+0000","last_clean":"2024-03-30T13:17:02.389568+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:17:02.389568+0000","last_undegraded":"2024-03-30T13:17:02.389568+0000","last_fullsized":"2024-03-30T13:17:02.389568+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:15:52.423297+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:15:52.423297+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:08.679223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":194,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-03-30T13:17:01.850929+0000","last_change":"2024-03-30T13:17:01.851001+0000","last_active":"2024-03-30T13:17:01.850929+0000","last_peered":"2024-03-30T13:17:01.850929+0000","last_clean":"2024-03-30T13:17:01.850929+0000","last_became_active":"2024-03-30T13:14:38.235512+0000","last_became_peered":"2024-03-30T13:14:38.235512+0000","last_unstale":"2024-03-30T13:17:01.850929+0000","last_undegraded":"2024-03-30T13:17:01.850929+0000","last_fullsized":"2024-03-30T13:17:01.850929+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:17:01.850742+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:17:01.850742+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:12.490730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":573,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-30T13:17:04.120973+0000","last_change":"2024-03-30T13:16:50.991946+0000","last_active":"2024-03-30T13:17:04.120973+0000","last_peered":"2024-03-30T13:17:04.120973+0000","last_clean":"2024-03-30T13:17:04.120973+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:17:04.120973+0000","last_undegraded":"2024-03-30T13:17:04.120973+0000","last_fullsized":"2024-03-30T13:17:04.120973+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:46.315690+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:16:46.315690+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:57.412865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":431732408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834205,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58456,"kb_used_data":22152,"kb_used_omap":5,"kb_used_meta":36282,"kb_avail":94313384,"statfs":{"total":96636764160,"available":96576905216,"internally_reserved":0,"allocated":22683648,"data_stored":18721417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":37152866},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157471,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64548,"kb_used_data":26896,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94307292,"statfs":{"total":96636764160,"available":96570667008,"internally_reserved":0,"allocated":27541504,"data_stored":23579273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":559,"seq":2400886718475,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56280,"kb_used_data":18636,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94315560,"statfs":{"total":96636764160,"available":96579133440,"internally_reserved":0,"allocated":19083264,"data_stored":15100630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12639,"internal_metadata":38522529},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509702,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48756,"kb_used_data":13416,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323084,"statfs":{"total":96636764160,"available":96586838016,"internally_reserved":0,"allocated":13737984,"data_stored":9748549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375998981,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49888,"kb_used_data":12692,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94321952,"statfs":{"total":96636764160,"available":96585678848,"internally_reserved":0,"allocated":12996608,"data_stored":9007173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509703,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51768,"kb_used_data":20580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320072,"statfs":{"total":96636764160,"available":96583753728,"internally_reserved":0,"allocated":21073920,"data_stored":17091286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297097,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54760,"kb_used_data":17112,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94317080,"statfs":{"total":96636764160,"available":96580689920,"internally_reserved":0,"allocated":17522688,"data_stored":13546855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147411,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51244,"kb_used_data":19544,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94320596,"statfs":{"total":96636764160,"available":96584290304,"internally_reserved":0,"allocated":20013056,"data_stored":16050825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:17:06.495 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:17:06.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:06.595+0000 7f9e9eff1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:06.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:06.607+0000 7f9e9eff1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:06.780 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:17:06.781 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:17:06.798 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1237,"stamp":"2024-03-30T13:17:06.156452+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":431732408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":435700,"kb_used_data":151028,"kb_used_omap":57,"kb_used_meta":284486,"kb_avail":754539020,"statfs":{"total":773094113280,"available":772647956480,"internally_reserved":0,"allocated":154652672,"data_stored":122846008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59250,"internal_metadata":291313806},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001015"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1392,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-03-30T13:17:00.605130+0000","last_change":"2024-03-30T13:16:50.692000+0000","last_active":"2024-03-30T13:17:00.605130+0000","last_peered":"2024-03-30T13:17:00.605130+0000","last_clean":"2024-03-30T13:17:00.605130+0000","last_became_active":"2024-03-30T13:16:20.316817+0000","last_became_peered":"2024-03-30T13:16:20.316817+0000","last_unstale":"2024-03-30T13:17:00.605130+0000","last_undegraded":"2024-03-30T13:17:00.605130+0000","last_fullsized":"2024-03-30T13:17:00.605130+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:50.691736+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:16:50.691736+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:51.654673+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":32117715,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1234,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-30T13:17:04.218942+0000","last_change":"2024-03-30T13:16:20.315116+0000","last_active":"2024-03-30T13:17:04.218942+0000","last_peered":"2024-03-30T13:17:04.218942+0000","last_clean":"2024-03-30T13:17:04.218942+0000","last_became_active":"2024-03-30T13:16:20.313939+0000","last_became_peered":"2024-03-30T13:16:20.313939+0000","last_unstale":"2024-03-30T13:17:04.218942+0000","last_undegraded":"2024-03-30T13:17:04.218942+0000","last_fullsized":"2024-03-30T13:17:04.218942+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:16:02.397699+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:16:02.397699+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:29.215054+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":949,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-30T13:17:02.389627+0000","last_change":"2024-03-30T13:16:50.994562+0000","last_active":"2024-03-30T13:17:02.389627+0000","last_peered":"2024-03-30T13:17:02.389627+0000","last_clean":"2024-03-30T13:17:02.389627+0000","last_became_active":"2024-03-30T13:16:50.993492+0000","last_became_peered":"2024-03-30T13:16:50.993492+0000","last_unstale":"2024-03-30T13:17:02.389627+0000","last_undegraded":"2024-03-30T13:17:02.389627+0000","last_fullsized":"2024-03-30T13:17:02.389627+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:16:04.484050+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:16:04.484050+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:08.749359+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":54706092,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":827,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-30T13:17:02.389560+0000","last_change":"2024-03-30T13:16:20.586918+0000","last_active":"2024-03-30T13:17:02.389560+0000","last_peered":"2024-03-30T13:17:02.389560+0000","last_clean":"2024-03-30T13:17:02.389560+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:17:02.389560+0000","last_undegraded":"2024-03-30T13:17:02.389560+0000","last_fullsized":"2024-03-30T13:17:02.389560+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:20.586668+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:16:20.586668+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:50.224020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":197,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-03-30T13:17:00.095015+0000","last_change":"2024-03-30T13:16:50.770031+0000","last_active":"2024-03-30T13:17:00.095015+0000","last_peered":"2024-03-30T13:17:00.095015+0000","last_clean":"2024-03-30T13:17:00.095015+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:17:00.095015+0000","last_undegraded":"2024-03-30T13:17:00.095015+0000","last_fullsized":"2024-03-30T13:17:00.095015+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:16:50.769668+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:16:50.769668+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:04.116739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1360,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-03-30T13:17:00.601897+0000","last_change":"2024-03-30T13:16:19.587276+0000","last_active":"2024-03-30T13:17:00.601897+0000","last_peered":"2024-03-30T13:17:00.601897+0000","last_clean":"2024-03-30T13:17:00.601897+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:17:00.601897+0000","last_undegraded":"2024-03-30T13:17:00.601897+0000","last_fullsized":"2024-03-30T13:17:00.601897+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:16:19.587123+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:16:19.587123+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:25.576107+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":922,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-30T13:17:02.389619+0000","last_change":"2024-03-30T13:16:57.724504+0000","last_active":"2024-03-30T13:17:02.389619+0000","last_peered":"2024-03-30T13:17:02.389619+0000","last_clean":"2024-03-30T13:17:02.389619+0000","last_became_active":"2024-03-30T13:16:20.312300+0000","last_became_peered":"2024-03-30T13:16:20.312300+0000","last_unstale":"2024-03-30T13:17:02.389619+0000","last_undegraded":"2024-03-30T13:17:02.389619+0000","last_fullsized":"2024-03-30T13:17:02.389619+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:16:57.724219+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:16:57.724219+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:23.534886+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58554796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":592,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-30T13:17:04.120980+0000","last_change":"2024-03-30T13:16:51.187305+0000","last_active":"2024-03-30T13:17:04.120980+0000","last_peered":"2024-03-30T13:17:04.120980+0000","last_clean":"2024-03-30T13:17:04.120980+0000","last_became_active":"2024-03-30T13:16:51.186350+0000","last_became_peered":"2024-03-30T13:16:51.186350+0000","last_unstale":"2024-03-30T13:17:04.120980+0000","last_undegraded":"2024-03-30T13:17:04.120980+0000","last_fullsized":"2024-03-30T13:17:04.120980+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:16:09.352215+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:16:09.352215+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:32.403615+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1468,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-03-30T13:17:00.601534+0000","last_change":"2024-03-30T13:16:45.572619+0000","last_active":"2024-03-30T13:17:00.601534+0000","last_peered":"2024-03-30T13:17:00.601534+0000","last_clean":"2024-03-30T13:17:00.601534+0000","last_became_active":"2024-03-30T13:14:38.238174+0000","last_became_peered":"2024-03-30T13:14:38.238174+0000","last_unstale":"2024-03-30T13:17:00.601534+0000","last_undegraded":"2024-03-30T13:17:00.601534+0000","last_fullsized":"2024-03-30T13:17:00.601534+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:16:45.572362+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:06.255142+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":71059072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1106,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-30T13:17:04.218630+0000","last_change":"2024-03-30T13:16:20.318505+0000","last_active":"2024-03-30T13:17:04.218630+0000","last_peered":"2024-03-30T13:17:04.218630+0000","last_clean":"2024-03-30T13:17:04.218630+0000","last_became_active":"2024-03-30T13:16:20.315944+0000","last_became_peered":"2024-03-30T13:16:20.315944+0000","last_unstale":"2024-03-30T13:17:04.218630+0000","last_undegraded":"2024-03-30T13:17:04.218630+0000","last_fullsized":"2024-03-30T13:17:04.218630+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_clean_scrub_stamp":"2024-03-30T13:16:16.623820+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:35.532032+0000","scrub_duration":659,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1376,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-03-30T13:17:00.605330+0000","last_change":"2024-03-30T13:16:32.758023+0000","last_active":"2024-03-30T13:17:00.605330+0000","last_peered":"2024-03-30T13:17:00.605330+0000","last_clean":"2024-03-30T13:17:00.605330+0000","last_became_active":"2024-03-30T13:16:20.313826+0000","last_became_peered":"2024-03-30T13:16:20.313826+0000","last_unstale":"2024-03-30T13:17:00.605330+0000","last_undegraded":"2024-03-30T13:17:00.605330+0000","last_fullsized":"2024-03-30T13:17:00.605330+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:16:32.757775+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:16:32.757775+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:50.377606+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":38218623,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1446,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-03-30T13:17:00.605453+0000","last_change":"2024-03-30T13:16:35.688621+0000","last_active":"2024-03-30T13:17:00.605453+0000","last_peered":"2024-03-30T13:17:00.605453+0000","last_clean":"2024-03-30T13:17:00.605453+0000","last_became_active":"2024-03-30T13:14:38.236424+0000","last_became_peered":"2024-03-30T13:14:38.236424+0000","last_unstale":"2024-03-30T13:17:00.605453+0000","last_undegraded":"2024-03-30T13:17:00.605453+0000","last_fullsized":"2024-03-30T13:17:00.605453+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:16:35.688353+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:16:35.688353+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:37.650028+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":76975004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":210,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-03-30T13:17:00.094961+0000","last_change":"2024-03-30T13:16:37.690115+0000","last_active":"2024-03-30T13:17:00.094961+0000","last_peered":"2024-03-30T13:17:00.094961+0000","last_clean":"2024-03-30T13:17:00.094961+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:17:00.094961+0000","last_undegraded":"2024-03-30T13:17:00.094961+0000","last_fullsized":"2024-03-30T13:17:00.094961+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:16:37.689862+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:16:37.689862+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:39.492459+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":798,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-30T13:17:02.389568+0000","last_change":"2024-03-30T13:15:52.423713+0000","last_active":"2024-03-30T13:17:02.389568+0000","last_peered":"2024-03-30T13:17:02.389568+0000","last_clean":"2024-03-30T13:17:02.389568+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:17:02.389568+0000","last_undegraded":"2024-03-30T13:17:02.389568+0000","last_fullsized":"2024-03-30T13:17:02.389568+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:15:52.423297+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:15:52.423297+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:08.679223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":194,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-03-30T13:17:01.850929+0000","last_change":"2024-03-30T13:17:01.851001+0000","last_active":"2024-03-30T13:17:01.850929+0000","last_peered":"2024-03-30T13:17:01.850929+0000","last_clean":"2024-03-30T13:17:01.850929+0000","last_became_active":"2024-03-30T13:14:38.235512+0000","last_became_peered":"2024-03-30T13:14:38.235512+0000","last_unstale":"2024-03-30T13:17:01.850929+0000","last_undegraded":"2024-03-30T13:17:01.850929+0000","last_fullsized":"2024-03-30T13:17:01.850929+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:17:01.850742+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:17:01.850742+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:12.490730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":573,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-30T13:17:04.120973+0000","last_change":"2024-03-30T13:16:50.991946+0000","last_active":"2024-03-30T13:17:04.120973+0000","last_peered":"2024-03-30T13:17:04.120973+0000","last_clean":"2024-03-30T13:17:04.120973+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:17:04.120973+0000","last_undegraded":"2024-03-30T13:17:04.120973+0000","last_fullsized":"2024-03-30T13:17:04.120973+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:46.315690+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:16:46.315690+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:57.412865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":431732408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834205,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58456,"kb_used_data":22152,"kb_used_omap":5,"kb_used_meta":36282,"kb_avail":94313384,"statfs":{"total":96636764160,"available":96576905216,"internally_reserved":0,"allocated":22683648,"data_stored":18721417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":37152866},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157471,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64548,"kb_used_data":26896,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94307292,"statfs":{"total":96636764160,"available":96570667008,"internally_reserved":0,"allocated":27541504,"data_stored":23579273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":559,"seq":2400886718475,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56280,"kb_used_data":18636,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94315560,"statfs":{"total":96636764160,"available":96579133440,"internally_reserved":0,"allocated":19083264,"data_stored":15100630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12639,"internal_metadata":38522529},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509702,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48756,"kb_used_data":13416,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323084,"statfs":{"total":96636764160,"available":96586838016,"internally_reserved":0,"allocated":13737984,"data_stored":9748549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375998981,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49888,"kb_used_data":12692,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94321952,"statfs":{"total":96636764160,"available":96585678848,"internally_reserved":0,"allocated":12996608,"data_stored":9007173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509703,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51768,"kb_used_data":20580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320072,"statfs":{"total":96636764160,"available":96583753728,"internally_reserved":0,"allocated":21073920,"data_stored":17091286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297097,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54760,"kb_used_data":17112,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94317080,"statfs":{"total":96636764160,"available":96580689920,"internally_reserved":0,"allocated":17522688,"data_stored":13546855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147411,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51244,"kb_used_data":19544,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94320596,"statfs":{"total":96636764160,"available":96584290304,"internally_reserved":0,"allocated":20013056,"data_stored":16050825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:17:06.799 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:17:06.799 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:17:06.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:06.911+0000 7f7b0adc3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:06.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:06.922+0000 7f7b0adc3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:07.100 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:17:07.101 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":597,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:17:06.136054+0000","last_up_change":"2024-03-30T13:16:49.955977+0000","last_in_change":"2024-03-30T13:16:49.955977+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":30,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":576,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":461,"last_clean_end":575,"up_from":581,"up_thru":581,"down_at":576,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6818","nonce":4030404750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6821","nonce":4030404750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6828","nonce":4030404750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6825","nonce":4030404750}]},"public_addr":"172.21.15.67:6818/4030404750","cluster_addr":"172.21.15.67:6821/4030404750","heartbeat_back_addr":"172.21.15.67:6828/4030404750","heartbeat_front_addr":"172.21.15.67:6825/4030404750","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":578,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":388,"up_from":393,"up_thru":515,"down_at":389,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6801","nonce":3837194718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6803","nonce":3837194718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6807","nonce":3837194718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6805","nonce":3837194718}]},"public_addr":"172.21.15.67:6801/3837194718","cluster_addr":"172.21.15.67:6803/3837194718","heartbeat_back_addr":"172.21.15.67:6807/3837194718","heartbeat_front_addr":"172.21.15.67:6805/3837194718","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":536,"last_clean_end":552,"up_from":559,"up_thru":578,"down_at":553,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3095958959},{"type":"v1","addr":"172.21.15.82:6803","nonce":3095958959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":3095958959},{"type":"v1","addr":"172.21.15.82:6807","nonce":3095958959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":3095958959},{"type":"v1","addr":"172.21.15.82:6815","nonce":3095958959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":3095958959},{"type":"v1","addr":"172.21.15.82:6811","nonce":3095958959}]},"public_addr":"172.21.15.82:6803/3095958959","cluster_addr":"172.21.15.82:6807/3095958959","heartbeat_back_addr":"172.21.15.82:6815/3095958959","heartbeat_front_addr":"172.21.15.82:6811/3095958959","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":488,"last_clean_end":506,"up_from":515,"up_thru":515,"down_at":507,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6817","nonce":292505453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6819","nonce":292505453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6823","nonce":292505453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6821","nonce":292505453}]},"public_addr":"172.21.15.82:6817/292505453","cluster_addr":"172.21.15.82:6819/292505453","heartbeat_back_addr":"172.21.15.82:6823/292505453","heartbeat_front_addr":"172.21.15.82:6821/292505453","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":227,"last_clean_end":363,"up_from":372,"up_thru":576,"down_at":364,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6801","nonce":189641108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6806","nonce":189641108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6814","nonce":189641108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6810","nonce":189641108}]},"public_addr":"172.21.15.82:6801/189641108","cluster_addr":"172.21.15.82:6806/189641108","heartbeat_back_addr":"172.21.15.82:6814/189641108","heartbeat_front_addr":"172.21.15.82:6810/189641108","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":298,"last_clean_end":409,"up_from":419,"up_thru":581,"down_at":410,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6825","nonce":2166599911}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6827","nonce":2166599911}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6831","nonce":2166599911}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":2166599911},{"type":"v1","addr":"172.21.15.82:6829","nonce":2166599911}]},"public_addr":"172.21.15.82:6825/2166599911","cluster_addr":"172.21.15.82:6827/2166599911","heartbeat_back_addr":"172.21.15.82:6831/2166599911","heartbeat_front_addr":"172.21.15.82:6829/2166599911","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:16:10.169160+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:16:29.677248+0000","dead_epoch":576},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:16:56.914201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:16:01.634626+0000","dead_epoch":389},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:16:59.641873+0000","dead_epoch":553},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:15:50.735114+0000","dead_epoch":507},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:16:53.943474+0000","dead_epoch":364},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:16:56.924723+0000","dead_epoch":410}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:17:07.118 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:17:07.119 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:17:07.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:07.222+0000 7f240da08640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:07.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:07.238+0000 7f2407fff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:07.417 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:17:07.436 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:17:07.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:07.550+0000 7efe5d002640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:07.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:07.570+0000 7efe5d002640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:07.745 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:17:07.745 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:17:07.762 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:17:07.762 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:17:07.762 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:17:07.763 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:17:07.763 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 6, 3, 7, 5, 4, 1] 2024-03-30T13:17:07.763 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:17:07.763 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:17:07.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:07.866+0000 7f77fbe90640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:07.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:07.878+0000 7f77fbe90640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:08.052 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:17:08.052 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:17:08.068 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1237,"stamp":"2024-03-30T13:17:06.156452+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":431732408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":435700,"kb_used_data":151028,"kb_used_omap":57,"kb_used_meta":284486,"kb_avail":754539020,"statfs":{"total":773094113280,"available":772647956480,"internally_reserved":0,"allocated":154652672,"data_stored":122846008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59250,"internal_metadata":291313806},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001015"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1392,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-03-30T13:17:00.605130+0000","last_change":"2024-03-30T13:16:50.692000+0000","last_active":"2024-03-30T13:17:00.605130+0000","last_peered":"2024-03-30T13:17:00.605130+0000","last_clean":"2024-03-30T13:17:00.605130+0000","last_became_active":"2024-03-30T13:16:20.316817+0000","last_became_peered":"2024-03-30T13:16:20.316817+0000","last_unstale":"2024-03-30T13:17:00.605130+0000","last_undegraded":"2024-03-30T13:17:00.605130+0000","last_fullsized":"2024-03-30T13:17:00.605130+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:50.691736+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:16:50.691736+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:51.654673+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":32117715,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1234,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-30T13:17:04.218942+0000","last_change":"2024-03-30T13:16:20.315116+0000","last_active":"2024-03-30T13:17:04.218942+0000","last_peered":"2024-03-30T13:17:04.218942+0000","last_clean":"2024-03-30T13:17:04.218942+0000","last_became_active":"2024-03-30T13:16:20.313939+0000","last_became_peered":"2024-03-30T13:16:20.313939+0000","last_unstale":"2024-03-30T13:17:04.218942+0000","last_undegraded":"2024-03-30T13:17:04.218942+0000","last_fullsized":"2024-03-30T13:17:04.218942+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:16:02.397699+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:16:02.397699+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:29.215054+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":949,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-30T13:17:02.389627+0000","last_change":"2024-03-30T13:16:50.994562+0000","last_active":"2024-03-30T13:17:02.389627+0000","last_peered":"2024-03-30T13:17:02.389627+0000","last_clean":"2024-03-30T13:17:02.389627+0000","last_became_active":"2024-03-30T13:16:50.993492+0000","last_became_peered":"2024-03-30T13:16:50.993492+0000","last_unstale":"2024-03-30T13:17:02.389627+0000","last_undegraded":"2024-03-30T13:17:02.389627+0000","last_fullsized":"2024-03-30T13:17:02.389627+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:16:04.484050+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:16:04.484050+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:08.749359+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":54706092,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":827,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-30T13:17:02.389560+0000","last_change":"2024-03-30T13:16:20.586918+0000","last_active":"2024-03-30T13:17:02.389560+0000","last_peered":"2024-03-30T13:17:02.389560+0000","last_clean":"2024-03-30T13:17:02.389560+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:17:02.389560+0000","last_undegraded":"2024-03-30T13:17:02.389560+0000","last_fullsized":"2024-03-30T13:17:02.389560+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:20.586668+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:16:20.586668+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:50.224020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":197,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-03-30T13:17:00.095015+0000","last_change":"2024-03-30T13:16:50.770031+0000","last_active":"2024-03-30T13:17:00.095015+0000","last_peered":"2024-03-30T13:17:00.095015+0000","last_clean":"2024-03-30T13:17:00.095015+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:17:00.095015+0000","last_undegraded":"2024-03-30T13:17:00.095015+0000","last_fullsized":"2024-03-30T13:17:00.095015+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:16:50.769668+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:16:50.769668+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:04.116739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1360,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-03-30T13:17:00.601897+0000","last_change":"2024-03-30T13:16:19.587276+0000","last_active":"2024-03-30T13:17:00.601897+0000","last_peered":"2024-03-30T13:17:00.601897+0000","last_clean":"2024-03-30T13:17:00.601897+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:17:00.601897+0000","last_undegraded":"2024-03-30T13:17:00.601897+0000","last_fullsized":"2024-03-30T13:17:00.601897+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:16:19.587123+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:16:19.587123+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:25.576107+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":922,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-30T13:17:02.389619+0000","last_change":"2024-03-30T13:16:57.724504+0000","last_active":"2024-03-30T13:17:02.389619+0000","last_peered":"2024-03-30T13:17:02.389619+0000","last_clean":"2024-03-30T13:17:02.389619+0000","last_became_active":"2024-03-30T13:16:20.312300+0000","last_became_peered":"2024-03-30T13:16:20.312300+0000","last_unstale":"2024-03-30T13:17:02.389619+0000","last_undegraded":"2024-03-30T13:17:02.389619+0000","last_fullsized":"2024-03-30T13:17:02.389619+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:16:57.724219+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:16:57.724219+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:23.534886+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58554796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":592,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-30T13:17:04.120980+0000","last_change":"2024-03-30T13:16:51.187305+0000","last_active":"2024-03-30T13:17:04.120980+0000","last_peered":"2024-03-30T13:17:04.120980+0000","last_clean":"2024-03-30T13:17:04.120980+0000","last_became_active":"2024-03-30T13:16:51.186350+0000","last_became_peered":"2024-03-30T13:16:51.186350+0000","last_unstale":"2024-03-30T13:17:04.120980+0000","last_undegraded":"2024-03-30T13:17:04.120980+0000","last_fullsized":"2024-03-30T13:17:04.120980+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:16:09.352215+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:16:09.352215+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:32.403615+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1468,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-03-30T13:17:00.601534+0000","last_change":"2024-03-30T13:16:45.572619+0000","last_active":"2024-03-30T13:17:00.601534+0000","last_peered":"2024-03-30T13:17:00.601534+0000","last_clean":"2024-03-30T13:17:00.601534+0000","last_became_active":"2024-03-30T13:14:38.238174+0000","last_became_peered":"2024-03-30T13:14:38.238174+0000","last_unstale":"2024-03-30T13:17:00.601534+0000","last_undegraded":"2024-03-30T13:17:00.601534+0000","last_fullsized":"2024-03-30T13:17:00.601534+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:16:45.572362+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:06.255142+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":71059072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1106,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-30T13:17:04.218630+0000","last_change":"2024-03-30T13:16:20.318505+0000","last_active":"2024-03-30T13:17:04.218630+0000","last_peered":"2024-03-30T13:17:04.218630+0000","last_clean":"2024-03-30T13:17:04.218630+0000","last_became_active":"2024-03-30T13:16:20.315944+0000","last_became_peered":"2024-03-30T13:16:20.315944+0000","last_unstale":"2024-03-30T13:17:04.218630+0000","last_undegraded":"2024-03-30T13:17:04.218630+0000","last_fullsized":"2024-03-30T13:17:04.218630+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_clean_scrub_stamp":"2024-03-30T13:16:16.623820+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:35.532032+0000","scrub_duration":659,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1376,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-03-30T13:17:00.605330+0000","last_change":"2024-03-30T13:16:32.758023+0000","last_active":"2024-03-30T13:17:00.605330+0000","last_peered":"2024-03-30T13:17:00.605330+0000","last_clean":"2024-03-30T13:17:00.605330+0000","last_became_active":"2024-03-30T13:16:20.313826+0000","last_became_peered":"2024-03-30T13:16:20.313826+0000","last_unstale":"2024-03-30T13:17:00.605330+0000","last_undegraded":"2024-03-30T13:17:00.605330+0000","last_fullsized":"2024-03-30T13:17:00.605330+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:16:32.757775+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:16:32.757775+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:50.377606+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":38218623,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1446,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-03-30T13:17:00.605453+0000","last_change":"2024-03-30T13:16:35.688621+0000","last_active":"2024-03-30T13:17:00.605453+0000","last_peered":"2024-03-30T13:17:00.605453+0000","last_clean":"2024-03-30T13:17:00.605453+0000","last_became_active":"2024-03-30T13:14:38.236424+0000","last_became_peered":"2024-03-30T13:14:38.236424+0000","last_unstale":"2024-03-30T13:17:00.605453+0000","last_undegraded":"2024-03-30T13:17:00.605453+0000","last_fullsized":"2024-03-30T13:17:00.605453+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:16:35.688353+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:16:35.688353+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:37.650028+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":76975004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":210,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-03-30T13:17:00.094961+0000","last_change":"2024-03-30T13:16:37.690115+0000","last_active":"2024-03-30T13:17:00.094961+0000","last_peered":"2024-03-30T13:17:00.094961+0000","last_clean":"2024-03-30T13:17:00.094961+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:17:00.094961+0000","last_undegraded":"2024-03-30T13:17:00.094961+0000","last_fullsized":"2024-03-30T13:17:00.094961+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:16:37.689862+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:16:37.689862+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:39.492459+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":798,"reported_epoch":593,"state":"active+clean","last_fresh":"2024-03-30T13:17:02.389568+0000","last_change":"2024-03-30T13:15:52.423713+0000","last_active":"2024-03-30T13:17:02.389568+0000","last_peered":"2024-03-30T13:17:02.389568+0000","last_clean":"2024-03-30T13:17:02.389568+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:17:02.389568+0000","last_undegraded":"2024-03-30T13:17:02.389568+0000","last_fullsized":"2024-03-30T13:17:02.389568+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:15:52.423297+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:15:52.423297+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:08.679223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":194,"reported_epoch":591,"state":"active+clean","last_fresh":"2024-03-30T13:17:01.850929+0000","last_change":"2024-03-30T13:17:01.851001+0000","last_active":"2024-03-30T13:17:01.850929+0000","last_peered":"2024-03-30T13:17:01.850929+0000","last_clean":"2024-03-30T13:17:01.850929+0000","last_became_active":"2024-03-30T13:14:38.235512+0000","last_became_peered":"2024-03-30T13:14:38.235512+0000","last_unstale":"2024-03-30T13:17:01.850929+0000","last_undegraded":"2024-03-30T13:17:01.850929+0000","last_fullsized":"2024-03-30T13:17:01.850929+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:17:01.850742+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:17:01.850742+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:12.490730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":573,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-30T13:17:04.120973+0000","last_change":"2024-03-30T13:16:50.991946+0000","last_active":"2024-03-30T13:17:04.120973+0000","last_peered":"2024-03-30T13:17:04.120973+0000","last_clean":"2024-03-30T13:17:04.120973+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:17:04.120973+0000","last_undegraded":"2024-03-30T13:17:04.120973+0000","last_fullsized":"2024-03-30T13:17:04.120973+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:46.315690+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:16:46.315690+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:57.412865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":431732408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834205,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58456,"kb_used_data":22152,"kb_used_omap":5,"kb_used_meta":36282,"kb_avail":94313384,"statfs":{"total":96636764160,"available":96576905216,"internally_reserved":0,"allocated":22683648,"data_stored":18721417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":37152866},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157471,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64548,"kb_used_data":26896,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94307292,"statfs":{"total":96636764160,"available":96570667008,"internally_reserved":0,"allocated":27541504,"data_stored":23579273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":559,"seq":2400886718475,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56280,"kb_used_data":18636,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94315560,"statfs":{"total":96636764160,"available":96579133440,"internally_reserved":0,"allocated":19083264,"data_stored":15100630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12639,"internal_metadata":38522529},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509702,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48756,"kb_used_data":13416,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323084,"statfs":{"total":96636764160,"available":96586838016,"internally_reserved":0,"allocated":13737984,"data_stored":9748549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375998981,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49888,"kb_used_data":12692,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94321952,"statfs":{"total":96636764160,"available":96585678848,"internally_reserved":0,"allocated":12996608,"data_stored":9007173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509703,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51768,"kb_used_data":20580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320072,"statfs":{"total":96636764160,"available":96583753728,"internally_reserved":0,"allocated":21073920,"data_stored":17091286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":419,"seq":1799591297097,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54760,"kb_used_data":17112,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94317080,"statfs":{"total":96636764160,"available":96580689920,"internally_reserved":0,"allocated":17522688,"data_stored":13546855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12634,"internal_metadata":38522534},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147411,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51244,"kb_used_data":19544,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94320596,"statfs":{"total":96636764160,"available":96584290304,"internally_reserved":0,"allocated":20013056,"data_stored":16050825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:17:08.069 INFO:tasks.thrashosds.thrasher:Choosing PG 1.d with acting set [7, 4, 1] 2024-03-30T13:17:08.069 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 2, 6, 3, 7, 5, 4, 1] 2024-03-30T13:17:08.069 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-30T13:17:08.070 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:17:08.123 INFO:tasks.ceph.osd.7:Stopped 2024-03-30T13:17:08.124 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 2, 6, 3, 7, 5, 4, 1] 2024-03-30T13:17:08.124 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 7 2024-03-30T13:17:08.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:08.238+0000 7ff97296e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:08.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:08.250+0000 7ff97296e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:08.437 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:17:08.434+0000 7fbb38f08640 -1 mon.a@0(leader).osd e598 definitely_dead 0 2024-03-30T13:17:09.180 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.7. 2024-03-30T13:17:09.205 INFO:tasks.thrashosds.thrasher:dead_osds=[7], live_osds=[0, 2, 6, 3, 5, 4, 1] 2024-03-30T13:17:09.206 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:17:09.206 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:17:09.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:09.314+0000 7ffb7fc47640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:09.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:09.326+0000 7ffb7fc47640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:09.505 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:17:09.506 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:17:09.524 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1240,"stamp":"2024-03-30T13:17:09.177807+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":431732408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":381104,"kb_used_data":134096,"kb_used_omap":45,"kb_used_meta":246866,"kb_avail":660221776,"statfs":{"total":676457349120,"available":676067098624,"internally_reserved":0,"allocated":137314304,"data_stored":109381458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":46616,"internal_metadata":252791272},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.021831"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1403,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-03-30T13:17:06.146294+0000","last_change":"2024-03-30T13:16:50.692000+0000","last_active":"2024-03-30T13:17:06.146294+0000","last_peered":"2024-03-30T13:17:06.146294+0000","last_clean":"2024-03-30T13:17:06.146294+0000","last_became_active":"2024-03-30T13:16:20.316817+0000","last_became_peered":"2024-03-30T13:16:20.316817+0000","last_unstale":"2024-03-30T13:17:06.146294+0000","last_undegraded":"2024-03-30T13:17:06.146294+0000","last_fullsized":"2024-03-30T13:17:06.146294+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:50.691736+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:16:50.691736+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:51.654673+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":32117715,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1234,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-30T13:17:04.218942+0000","last_change":"2024-03-30T13:16:20.315116+0000","last_active":"2024-03-30T13:17:04.218942+0000","last_peered":"2024-03-30T13:17:04.218942+0000","last_clean":"2024-03-30T13:17:04.218942+0000","last_became_active":"2024-03-30T13:16:20.313939+0000","last_became_peered":"2024-03-30T13:16:20.313939+0000","last_unstale":"2024-03-30T13:17:04.218942+0000","last_undegraded":"2024-03-30T13:17:04.218942+0000","last_fullsized":"2024-03-30T13:17:04.218942+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:16:02.397699+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:16:02.397699+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:29.215054+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":949,"reported_epoch":593,"state":"stale+active+clean","last_fresh":"2024-03-30T13:17:02.389627+0000","last_change":"2024-03-30T13:16:50.994562+0000","last_active":"2024-03-30T13:17:02.389627+0000","last_peered":"2024-03-30T13:17:02.389627+0000","last_clean":"2024-03-30T13:17:02.389627+0000","last_became_active":"2024-03-30T13:16:50.993492+0000","last_became_peered":"2024-03-30T13:16:50.993492+0000","last_unstale":"2024-03-30T13:17:08.172783+0000","last_undegraded":"2024-03-30T13:17:02.389627+0000","last_fullsized":"2024-03-30T13:17:02.389627+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:16:04.484050+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:16:04.484050+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:08.749359+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":54706092,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":827,"reported_epoch":593,"state":"stale+active+clean","last_fresh":"2024-03-30T13:17:02.389560+0000","last_change":"2024-03-30T13:16:20.586918+0000","last_active":"2024-03-30T13:17:02.389560+0000","last_peered":"2024-03-30T13:17:02.389560+0000","last_clean":"2024-03-30T13:17:02.389560+0000","last_became_active":"2024-03-30T13:11:09.211342+0000","last_became_peered":"2024-03-30T13:11:09.211342+0000","last_unstale":"2024-03-30T13:17:08.172786+0000","last_undegraded":"2024-03-30T13:17:02.389560+0000","last_fullsized":"2024-03-30T13:17:02.389560+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:20.586668+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:16:20.586668+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:50.224020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":208,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-03-30T13:17:06.319553+0000","last_change":"2024-03-30T13:16:50.770031+0000","last_active":"2024-03-30T13:17:06.319553+0000","last_peered":"2024-03-30T13:17:06.319553+0000","last_clean":"2024-03-30T13:17:06.319553+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:17:06.319553+0000","last_undegraded":"2024-03-30T13:17:06.319553+0000","last_fullsized":"2024-03-30T13:17:06.319553+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:16:50.769668+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:16:50.769668+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:04.116739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1369,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-30T13:17:05.706807+0000","last_change":"2024-03-30T13:16:19.587276+0000","last_active":"2024-03-30T13:17:05.706807+0000","last_peered":"2024-03-30T13:17:05.706807+0000","last_clean":"2024-03-30T13:17:05.706807+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:17:05.706807+0000","last_undegraded":"2024-03-30T13:17:05.706807+0000","last_fullsized":"2024-03-30T13:17:05.706807+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:16:19.587123+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:16:19.587123+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:25.576107+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":922,"reported_epoch":593,"state":"stale+active+clean","last_fresh":"2024-03-30T13:17:02.389619+0000","last_change":"2024-03-30T13:16:57.724504+0000","last_active":"2024-03-30T13:17:02.389619+0000","last_peered":"2024-03-30T13:17:02.389619+0000","last_clean":"2024-03-30T13:17:02.389619+0000","last_became_active":"2024-03-30T13:16:20.312300+0000","last_became_peered":"2024-03-30T13:16:20.312300+0000","last_unstale":"2024-03-30T13:17:08.172788+0000","last_undegraded":"2024-03-30T13:17:02.389619+0000","last_fullsized":"2024-03-30T13:17:02.389619+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:16:57.724219+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:16:57.724219+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:23.534886+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58554796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":592,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-30T13:17:04.120980+0000","last_change":"2024-03-30T13:16:51.187305+0000","last_active":"2024-03-30T13:17:04.120980+0000","last_peered":"2024-03-30T13:17:04.120980+0000","last_clean":"2024-03-30T13:17:04.120980+0000","last_became_active":"2024-03-30T13:16:51.186350+0000","last_became_peered":"2024-03-30T13:16:51.186350+0000","last_unstale":"2024-03-30T13:17:04.120980+0000","last_undegraded":"2024-03-30T13:17:04.120980+0000","last_fullsized":"2024-03-30T13:17:04.120980+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:16:09.352215+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:16:09.352215+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:32.403615+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1477,"reported_epoch":596,"state":"active+clean","last_fresh":"2024-03-30T13:17:05.706681+0000","last_change":"2024-03-30T13:16:45.572619+0000","last_active":"2024-03-30T13:17:05.706681+0000","last_peered":"2024-03-30T13:17:05.706681+0000","last_clean":"2024-03-30T13:17:05.706681+0000","last_became_active":"2024-03-30T13:14:38.238174+0000","last_became_peered":"2024-03-30T13:14:38.238174+0000","last_unstale":"2024-03-30T13:17:05.706681+0000","last_undegraded":"2024-03-30T13:17:05.706681+0000","last_fullsized":"2024-03-30T13:17:05.706681+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:16:45.572362+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:06.255142+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":71059072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1106,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-30T13:17:04.218630+0000","last_change":"2024-03-30T13:16:20.318505+0000","last_active":"2024-03-30T13:17:04.218630+0000","last_peered":"2024-03-30T13:17:04.218630+0000","last_clean":"2024-03-30T13:17:04.218630+0000","last_became_active":"2024-03-30T13:16:20.315944+0000","last_became_peered":"2024-03-30T13:16:20.315944+0000","last_unstale":"2024-03-30T13:17:04.218630+0000","last_undegraded":"2024-03-30T13:17:04.218630+0000","last_fullsized":"2024-03-30T13:17:04.218630+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_clean_scrub_stamp":"2024-03-30T13:16:16.623820+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:35.532032+0000","scrub_duration":659,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1387,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-03-30T13:17:06.145969+0000","last_change":"2024-03-30T13:16:32.758023+0000","last_active":"2024-03-30T13:17:06.145969+0000","last_peered":"2024-03-30T13:17:06.145969+0000","last_clean":"2024-03-30T13:17:06.145969+0000","last_became_active":"2024-03-30T13:16:20.313826+0000","last_became_peered":"2024-03-30T13:16:20.313826+0000","last_unstale":"2024-03-30T13:17:06.145969+0000","last_undegraded":"2024-03-30T13:17:06.145969+0000","last_fullsized":"2024-03-30T13:17:06.145969+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:16:32.757775+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:16:32.757775+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:50.377606+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":38218623,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1457,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-03-30T13:17:06.146035+0000","last_change":"2024-03-30T13:16:35.688621+0000","last_active":"2024-03-30T13:17:06.146035+0000","last_peered":"2024-03-30T13:17:06.146035+0000","last_clean":"2024-03-30T13:17:06.146035+0000","last_became_active":"2024-03-30T13:14:38.236424+0000","last_became_peered":"2024-03-30T13:14:38.236424+0000","last_unstale":"2024-03-30T13:17:06.146035+0000","last_undegraded":"2024-03-30T13:17:06.146035+0000","last_fullsized":"2024-03-30T13:17:06.146035+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:16:35.688353+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:16:35.688353+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:37.650028+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":76975004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":221,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-03-30T13:17:06.319517+0000","last_change":"2024-03-30T13:16:37.690115+0000","last_active":"2024-03-30T13:17:06.319517+0000","last_peered":"2024-03-30T13:17:06.319517+0000","last_clean":"2024-03-30T13:17:06.319517+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:17:06.319517+0000","last_undegraded":"2024-03-30T13:17:06.319517+0000","last_fullsized":"2024-03-30T13:17:06.319517+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:16:37.689862+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:16:37.689862+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:39.492459+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":798,"reported_epoch":593,"state":"stale+active+clean","last_fresh":"2024-03-30T13:17:02.389568+0000","last_change":"2024-03-30T13:15:52.423713+0000","last_active":"2024-03-30T13:17:02.389568+0000","last_peered":"2024-03-30T13:17:02.389568+0000","last_clean":"2024-03-30T13:17:02.389568+0000","last_became_active":"2024-03-30T13:11:09.209381+0000","last_became_peered":"2024-03-30T13:11:09.209381+0000","last_unstale":"2024-03-30T13:17:08.172791+0000","last_undegraded":"2024-03-30T13:17:02.389568+0000","last_fullsized":"2024-03-30T13:17:02.389568+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:15:52.423297+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:15:52.423297+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:08.679223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":205,"reported_epoch":597,"state":"active+clean","last_fresh":"2024-03-30T13:17:06.319553+0000","last_change":"2024-03-30T13:17:01.851001+0000","last_active":"2024-03-30T13:17:06.319553+0000","last_peered":"2024-03-30T13:17:06.319553+0000","last_clean":"2024-03-30T13:17:06.319553+0000","last_became_active":"2024-03-30T13:14:38.235512+0000","last_became_peered":"2024-03-30T13:14:38.235512+0000","last_unstale":"2024-03-30T13:17:06.319553+0000","last_undegraded":"2024-03-30T13:17:06.319553+0000","last_fullsized":"2024-03-30T13:17:06.319553+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:17:01.850742+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:17:01.850742+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:12.490730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":573,"reported_epoch":595,"state":"active+clean","last_fresh":"2024-03-30T13:17:04.120973+0000","last_change":"2024-03-30T13:16:50.991946+0000","last_active":"2024-03-30T13:17:04.120973+0000","last_peered":"2024-03-30T13:17:04.120973+0000","last_clean":"2024-03-30T13:17:04.120973+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:17:04.120973+0000","last_undegraded":"2024-03-30T13:17:04.120973+0000","last_fullsized":"2024-03-30T13:17:04.120973+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:46.315690+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:16:46.315690+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:57.412865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":431732408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834206,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58504,"kb_used_data":22200,"kb_used_omap":5,"kb_used_meta":36282,"kb_avail":94313336,"statfs":{"total":96636764160,"available":96576856064,"internally_reserved":0,"allocated":22732800,"data_stored":18743365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":37152866},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157472,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64612,"kb_used_data":26968,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94307228,"statfs":{"total":96636764160,"available":96570601472,"internally_reserved":0,"allocated":27615232,"data_stored":23612195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":559,"seq":2400886718475,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56280,"kb_used_data":18636,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94315560,"statfs":{"total":96636764160,"available":96579133440,"internally_reserved":0,"allocated":19083264,"data_stored":15100630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12639,"internal_metadata":38522529},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509702,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48756,"kb_used_data":13416,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323084,"statfs":{"total":96636764160,"available":96586838016,"internally_reserved":0,"allocated":13737984,"data_stored":9748549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375998981,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49888,"kb_used_data":12692,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94321952,"statfs":{"total":96636764160,"available":96585678848,"internally_reserved":0,"allocated":12996608,"data_stored":9007173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509703,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51768,"kb_used_data":20580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320072,"statfs":{"total":96636764160,"available":96583753728,"internally_reserved":0,"allocated":21073920,"data_stored":17091286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147412,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51296,"kb_used_data":19604,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94320544,"statfs":{"total":96636764160,"available":96584237056,"internally_reserved":0,"allocated":20074496,"data_stored":16078260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:17:09.525 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:17:34.526 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:17:34.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:17:34.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:34.654+0000 7f6d5a193640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:34.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:34.666+0000 7f6d5a193640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:34.841 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:17:34.842 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:17:34.859 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1256,"stamp":"2024-03-30T13:17:34.164668+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":371362456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":391716,"kb_used_data":144716,"kb_used_omap":45,"kb_used_meta":246866,"kb_avail":660211164,"statfs":{"total":676457349120,"available":676056231936,"internally_reserved":0,"allocated":148189184,"data_stored":119932802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":46616,"internal_metadata":252791272},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003362"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1412,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-03-30T13:17:18.263064+0000","last_change":"2024-03-30T13:16:50.692000+0000","last_active":"2024-03-30T13:17:18.263064+0000","last_peered":"2024-03-30T13:17:18.263064+0000","last_clean":"2024-03-30T13:17:18.263064+0000","last_became_active":"2024-03-30T13:16:20.316817+0000","last_became_peered":"2024-03-30T13:16:20.316817+0000","last_unstale":"2024-03-30T13:17:18.263064+0000","last_undegraded":"2024-03-30T13:17:18.263064+0000","last_fullsized":"2024-03-30T13:17:18.263064+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:50.691736+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:04:16.526295+0000","last_clean_scrub_stamp":"2024-03-30T13:16:50.691736+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:51.654673+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":32117715,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1257,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-03-30T13:17:18.264063+0000","last_change":"2024-03-30T13:17:10.235185+0000","last_active":"2024-03-30T13:17:18.264063+0000","last_peered":"2024-03-30T13:17:18.264063+0000","last_clean":"2024-03-30T13:17:18.264063+0000","last_became_active":"2024-03-30T13:17:10.201730+0000","last_became_peered":"2024-03-30T13:17:10.201730+0000","last_unstale":"2024-03-30T13:17:18.264063+0000","last_undegraded":"2024-03-30T13:17:18.264063+0000","last_fullsized":"2024-03-30T13:17:18.264063+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:16:02.397699+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:16:02.397699+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:26.438838+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":31,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-03-30T13:17:18.851780+0000","last_change":"2024-03-30T13:17:13.219243+0000","last_active":"2024-03-30T13:17:18.851780+0000","last_peered":"2024-03-30T13:17:18.851780+0000","last_clean":"2024-03-30T13:17:18.851780+0000","last_became_active":"2024-03-30T13:17:10.201436+0000","last_became_peered":"2024-03-30T13:17:10.201436+0000","last_unstale":"2024-03-30T13:17:18.851780+0000","last_undegraded":"2024-03-30T13:17:18.851780+0000","last_fullsized":"2024-03-30T13:17:18.851780+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:17:13.218985+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:17:13.218985+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:27.344642+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":21,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-03-30T13:17:18.851210+0000","last_change":"2024-03-30T13:17:10.255507+0000","last_active":"2024-03-30T13:17:18.851210+0000","last_peered":"2024-03-30T13:17:18.851210+0000","last_clean":"2024-03-30T13:17:18.851210+0000","last_became_active":"2024-03-30T13:17:10.200130+0000","last_became_peered":"2024-03-30T13:17:10.200130+0000","last_unstale":"2024-03-30T13:17:18.851210+0000","last_undegraded":"2024-03-30T13:17:18.851210+0000","last_fullsized":"2024-03-30T13:17:18.851210+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:20.586668+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:16:20.586668+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:43.612734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":218,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-03-30T13:17:18.854462+0000","last_change":"2024-03-30T13:16:50.770031+0000","last_active":"2024-03-30T13:17:18.854462+0000","last_peered":"2024-03-30T13:17:18.854462+0000","last_clean":"2024-03-30T13:17:18.854462+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:17:18.854462+0000","last_undegraded":"2024-03-30T13:17:18.854462+0000","last_fullsized":"2024-03-30T13:17:18.854462+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:16:50.769668+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:16:50.769668+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:04.116739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1388,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-03-30T13:17:28.727385+0000","last_change":"2024-03-30T13:17:28.727465+0000","last_active":"2024-03-30T13:17:28.727385+0000","last_peered":"2024-03-30T13:17:28.727385+0000","last_clean":"2024-03-30T13:17:28.727385+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:17:28.727385+0000","last_undegraded":"2024-03-30T13:17:28.727385+0000","last_fullsized":"2024-03-30T13:17:28.727385+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:17:28.727219+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:17:28.727219+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:37.241722+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":36,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-03-30T13:17:18.263915+0000","last_change":"2024-03-30T13:17:10.572870+0000","last_active":"2024-03-30T13:17:18.263915+0000","last_peered":"2024-03-30T13:17:18.263915+0000","last_clean":"2024-03-30T13:17:18.263915+0000","last_became_active":"2024-03-30T13:17:10.203539+0000","last_became_peered":"2024-03-30T13:17:10.203539+0000","last_unstale":"2024-03-30T13:17:18.263915+0000","last_undegraded":"2024-03-30T13:17:18.263915+0000","last_fullsized":"2024-03-30T13:17:18.263915+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:16:57.724219+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:16:57.724219+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:05.526615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":606,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-03-30T13:17:18.851244+0000","last_change":"2024-03-30T13:16:51.187305+0000","last_active":"2024-03-30T13:17:18.851244+0000","last_peered":"2024-03-30T13:17:18.851244+0000","last_clean":"2024-03-30T13:17:18.851244+0000","last_became_active":"2024-03-30T13:16:51.186350+0000","last_became_peered":"2024-03-30T13:16:51.186350+0000","last_unstale":"2024-03-30T13:17:18.851244+0000","last_undegraded":"2024-03-30T13:17:18.851244+0000","last_fullsized":"2024-03-30T13:17:18.851244+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:16:09.352215+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:16:09.352215+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:32.403615+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1499,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-03-30T13:17:18.852277+0000","last_change":"2024-03-30T13:17:11.101681+0000","last_active":"2024-03-30T13:17:18.852277+0000","last_peered":"2024-03-30T13:17:18.852277+0000","last_clean":"2024-03-30T13:17:18.852277+0000","last_became_active":"2024-03-30T13:17:10.202275+0000","last_became_peered":"2024-03-30T13:17:10.202275+0000","last_unstale":"2024-03-30T13:17:18.852277+0000","last_undegraded":"2024-03-30T13:17:18.852277+0000","last_fullsized":"2024-03-30T13:17:18.852277+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:16:45.572362+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:08.570916+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":76134720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1120,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-03-30T13:17:18.263898+0000","last_change":"2024-03-30T13:16:20.318505+0000","last_active":"2024-03-30T13:17:18.263898+0000","last_peered":"2024-03-30T13:17:18.263898+0000","last_clean":"2024-03-30T13:17:18.263898+0000","last_became_active":"2024-03-30T13:16:20.315944+0000","last_became_peered":"2024-03-30T13:16:20.315944+0000","last_unstale":"2024-03-30T13:17:18.263898+0000","last_undegraded":"2024-03-30T13:17:18.263898+0000","last_fullsized":"2024-03-30T13:17:18.263898+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_clean_scrub_stamp":"2024-03-30T13:16:16.623820+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:35.532032+0000","scrub_duration":659,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1396,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-03-30T13:17:18.262599+0000","last_change":"2024-03-30T13:16:32.758023+0000","last_active":"2024-03-30T13:17:18.262599+0000","last_peered":"2024-03-30T13:17:18.262599+0000","last_clean":"2024-03-30T13:17:18.262599+0000","last_became_active":"2024-03-30T13:16:20.313826+0000","last_became_peered":"2024-03-30T13:16:20.313826+0000","last_unstale":"2024-03-30T13:17:18.262599+0000","last_undegraded":"2024-03-30T13:17:18.262599+0000","last_fullsized":"2024-03-30T13:17:18.262599+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:16:32.757775+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:26.917553+0000","last_clean_scrub_stamp":"2024-03-30T13:16:32.757775+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:50.377606+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":38218623,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1466,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-03-30T13:17:18.262439+0000","last_change":"2024-03-30T13:16:35.688621+0000","last_active":"2024-03-30T13:17:18.262439+0000","last_peered":"2024-03-30T13:17:18.262439+0000","last_clean":"2024-03-30T13:17:18.262439+0000","last_became_active":"2024-03-30T13:14:38.236424+0000","last_became_peered":"2024-03-30T13:14:38.236424+0000","last_unstale":"2024-03-30T13:17:18.262439+0000","last_undegraded":"2024-03-30T13:17:18.262439+0000","last_fullsized":"2024-03-30T13:17:18.262439+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:16:35.688353+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:16:35.688353+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:37.650028+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":76975004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":231,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-03-30T13:17:18.854402+0000","last_change":"2024-03-30T13:16:37.690115+0000","last_active":"2024-03-30T13:17:18.854402+0000","last_peered":"2024-03-30T13:17:18.854402+0000","last_clean":"2024-03-30T13:17:18.854402+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:17:18.854402+0000","last_undegraded":"2024-03-30T13:17:18.854402+0000","last_fullsized":"2024-03-30T13:17:18.854402+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:16:37.689862+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:16:37.689862+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:39.492459+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":27,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-03-30T13:17:19.945352+0000","last_change":"2024-03-30T13:17:19.945418+0000","last_active":"2024-03-30T13:17:19.945352+0000","last_peered":"2024-03-30T13:17:19.945352+0000","last_clean":"2024-03-30T13:17:19.945352+0000","last_became_active":"2024-03-30T13:17:10.207995+0000","last_became_peered":"2024-03-30T13:17:10.207995+0000","last_unstale":"2024-03-30T13:17:19.945352+0000","last_undegraded":"2024-03-30T13:17:19.945352+0000","last_fullsized":"2024-03-30T13:17:19.945352+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:17:19.945177+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:17:19.945177+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:32.456662+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":215,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-03-30T13:17:18.854309+0000","last_change":"2024-03-30T13:17:01.851001+0000","last_active":"2024-03-30T13:17:18.854309+0000","last_peered":"2024-03-30T13:17:18.854309+0000","last_clean":"2024-03-30T13:17:18.854309+0000","last_became_active":"2024-03-30T13:14:38.235512+0000","last_became_peered":"2024-03-30T13:14:38.235512+0000","last_unstale":"2024-03-30T13:17:18.854309+0000","last_undegraded":"2024-03-30T13:17:18.854309+0000","last_fullsized":"2024-03-30T13:17:18.854309+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:17:01.850742+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:17:01.850742+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:12.490730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":587,"reported_epoch":602,"state":"active+clean","last_fresh":"2024-03-30T13:17:18.852848+0000","last_change":"2024-03-30T13:16:50.991946+0000","last_active":"2024-03-30T13:17:18.852848+0000","last_peered":"2024-03-30T13:17:18.852848+0000","last_clean":"2024-03-30T13:17:18.852848+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:17:18.852848+0000","last_undegraded":"2024-03-30T13:17:18.852848+0000","last_fullsized":"2024-03-30T13:17:18.852848+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:46.315690+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:16:46.315690+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:57.412865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":371362456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":107036672,"data_stored":107036672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834211,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58584,"kb_used_data":22284,"kb_used_omap":5,"kb_used_meta":36282,"kb_avail":94313256,"statfs":{"total":96636764160,"available":96576774144,"internally_reserved":0,"allocated":22818816,"data_stored":18782190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":37152866},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157477,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65312,"kb_used_data":27668,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94306528,"statfs":{"total":96636764160,"available":96569884672,"internally_reserved":0,"allocated":28332032,"data_stored":24295406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":559,"seq":2400886718480,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56376,"kb_used_data":18732,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94315464,"statfs":{"total":96636764160,"available":96579035136,"internally_reserved":0,"allocated":19181568,"data_stored":15144942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12639,"internal_metadata":38522529},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509707,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54940,"kb_used_data":19596,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94316900,"statfs":{"total":96636764160,"available":96580505600,"internally_reserved":0,"allocated":20066304,"data_stored":16029678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375998986,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53280,"kb_used_data":16084,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94318560,"statfs":{"total":96636764160,"available":96582205440,"internally_reserved":0,"allocated":16470016,"data_stored":12433390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509708,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51860,"kb_used_data":20676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94319980,"statfs":{"total":96636764160,"available":96583659520,"internally_reserved":0,"allocated":21172224,"data_stored":17135598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147417,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51364,"kb_used_data":19676,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94320476,"statfs":{"total":96636764160,"available":96584167424,"internally_reserved":0,"allocated":20148224,"data_stored":16111598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9121792,"data_stored":9121792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12718080,"data_stored":12718080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20983808,"data_stored":20983808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:17:34.860 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:17:34.860 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:17:34.860 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2024-03-30T13:17:34.860 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-30T13:17:34.860 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-30T13:17:34.863 INFO:tasks.ceph.osd.7:Started 2024-03-30T13:17:34.863 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-30T13:17:34.933 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:17:34.935+0000 7fb8d85487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:34.940 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:17:34.943+0000 7fb8d85487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:34.940 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:17:34.943+0000 7fb8d85487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:34.980 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T13:17:34.980 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T13:17:34.980 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T13:17:34.980 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T13:17:34.980 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T13:17:34.980 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T13:17:34.980 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-03-30T13:17:34.980 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-03-30T13:17:34.980 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-03-30T13:17:34.980 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-03-30T13:17:34.980 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-03-30T13:17:34.981 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T13:17:34.982 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:17:34.983 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-03-30T13:17:35.484 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:17:35.486+0000 7fb8d85487c0 -1 Falling back to public interface 2024-03-30T13:17:39.285 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:17:39.286+0000 7fb8d85487c0 -1 osd.7 597 log_to_monitors true 2024-03-30T13:17:39.359 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:17:39.362+0000 7fb8d031d640 -1 osd.7 597 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:17:39.984 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-30T13:17:40.111 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T13:17:40.111 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T13:17:40.112 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T13:17:40.112 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T13:17:40.125 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:17:40.125 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-03-30T13:17:40.125 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 7 2024-03-30T13:17:40.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:40.254+0000 7f31d4e20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:40.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:40.266+0000 7f31d4e20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:40.460 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:17:40.458+0000 7fbb38f08640 -1 mon.a@0(leader).osd e604 definitely_dead 0 2024-03-30T13:17:41.328 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.7. 2024-03-30T13:17:41.359 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-03-30T13:17:41.359 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:17:41.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:41.474+0000 7fe3f04ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:41.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:41.490+0000 7fe3f04ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:41.899 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:17:41.928 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:17:42.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:42.042+0000 7f064c2ab640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:42.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:42.058+0000 7f064c2ab640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:42.919 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:17:42.947 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:17:43.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:43.066+0000 7fcaadb9b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:43.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:43.082+0000 7fcaadb9b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:43.909 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:17:43.939 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:17:44.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:44.057+0000 7fe8ba12d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:44.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:44.073+0000 7fe8ba12d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:44.919 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:17:44.950 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:17:45.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:45.065+0000 7f374e41b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:45.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:45.077+0000 7f374e41b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:45.921 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:17:45.944 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:17:46.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:46.065+0000 7fb3d2218640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:46.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:46.081+0000 7fb3d2218640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:46.928 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:17:46.951 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:17:47.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:47.069+0000 7f860525e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:47.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:47.085+0000 7f860525e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:47.941 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:17:47.968 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:17:48.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:48.089+0000 7fda3e66a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:48.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:48.105+0000 7fda3e66a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:48.961 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:17:48.988 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:17:49.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:49.105+0000 7f99d9247640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:49.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:49.121+0000 7f99d9247640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:49.964 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:17:49.989 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:17:50.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:50.105+0000 7fcbb303c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:50.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:50.125+0000 7fcbb303c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:50.974 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:17:51.000 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:17:51.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:51.117+0000 7f1241695640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:51.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:51.133+0000 7f1241695640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:51.981 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:17:52.005 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:17:52.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:52.125+0000 7feac3ce8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:52.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:52.141+0000 7feac3ce8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:52.986 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:17:53.010 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:17:53.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:53.125+0000 7f5df913c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:53.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:53.141+0000 7f5df913c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:53.996 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:17:54.025 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:17:54.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:54.145+0000 7fc8046fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:54.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:54.161+0000 7fc8046fc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:55.014 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:17:55.039 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:17:55.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:55.157+0000 7f4d2ad1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:55.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:55.173+0000 7f4d2ad1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:56.016 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:17:56.046 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:17:56.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:56.165+0000 7f1ed70b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:56.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:56.181+0000 7f1ed70b0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:57.046 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:17:57.070 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:17:57.070 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:17:57.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:57.189+0000 7f6ea4f04640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:57.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:57.201+0000 7f6ea4f04640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:57.379 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:17:57.380 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:17:57.398 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1286,"stamp":"2024-03-30T13:17:57.044007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":346774145,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":49,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":442752,"kb_used_data":153940,"kb_used_omap":59,"kb_used_meta":288644,"kb_avail":754531968,"statfs":{"total":773094113280,"available":772640735232,"internally_reserved":0,"allocated":157634560,"data_stored":124547576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60898,"internal_metadata":295571998},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.051988"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1452,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.861306+0000","last_change":"2024-03-30T13:17:53.569444+0000","last_active":"2024-03-30T13:17:54.861306+0000","last_peered":"2024-03-30T13:17:54.861306+0000","last_clean":"2024-03-30T13:17:54.861306+0000","last_became_active":"2024-03-30T13:16:20.316817+0000","last_became_peered":"2024-03-30T13:16:20.316817+0000","last_unstale":"2024-03-30T13:17:54.861306+0000","last_undegraded":"2024-03-30T13:17:54.861306+0000","last_fullsized":"2024-03-30T13:17:54.861306+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:17:53.569165+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:07.627928+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":32117715,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1314,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.004283+0000","last_change":"2024-03-30T13:17:44.939037+0000","last_active":"2024-03-30T13:17:54.004283+0000","last_peered":"2024-03-30T13:17:54.004283+0000","last_clean":"2024-03-30T13:17:54.004283+0000","last_became_active":"2024-03-30T13:17:44.937731+0000","last_became_peered":"2024-03-30T13:17:44.937731+0000","last_unstale":"2024-03-30T13:17:54.004283+0000","last_undegraded":"2024-03-30T13:17:54.004283+0000","last_fullsized":"2024-03-30T13:17:54.004283+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:17:32.544446+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:17:32.544446+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:33.043891+0000","scrub_duration":331,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":80,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.861339+0000","last_change":"2024-03-30T13:17:44.938352+0000","last_active":"2024-03-30T13:17:54.861339+0000","last_peered":"2024-03-30T13:17:54.861339+0000","last_clean":"2024-03-30T13:17:54.861339+0000","last_became_active":"2024-03-30T13:17:44.937300+0000","last_became_peered":"2024-03-30T13:17:44.937300+0000","last_unstale":"2024-03-30T13:17:54.861339+0000","last_undegraded":"2024-03-30T13:17:54.861339+0000","last_fullsized":"2024-03-30T13:17:54.861339+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:17:13.218985+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:17:13.218985+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:22.528329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":76,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.860970+0000","last_change":"2024-03-30T13:17:46.411658+0000","last_active":"2024-03-30T13:17:54.860970+0000","last_peered":"2024-03-30T13:17:54.860970+0000","last_clean":"2024-03-30T13:17:54.860970+0000","last_became_active":"2024-03-30T13:17:44.934742+0000","last_became_peered":"2024-03-30T13:17:44.934742+0000","last_unstale":"2024-03-30T13:17:54.860970+0000","last_undegraded":"2024-03-30T13:17:54.860970+0000","last_fullsized":"2024-03-30T13:17:54.860970+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:17:46.411360+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:17:46.411360+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:47.789848+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":243,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-03-30T13:17:51.397127+0000","last_change":"2024-03-30T13:16:50.770031+0000","last_active":"2024-03-30T13:17:51.397127+0000","last_peered":"2024-03-30T13:17:51.397127+0000","last_clean":"2024-03-30T13:17:51.397127+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:17:51.397127+0000","last_undegraded":"2024-03-30T13:17:51.397127+0000","last_fullsized":"2024-03-30T13:17:51.397127+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:16:50.769668+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:16:50.769668+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:04.116739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1413,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-03-30T13:17:51.394431+0000","last_change":"2024-03-30T13:17:28.727465+0000","last_active":"2024-03-30T13:17:51.394431+0000","last_peered":"2024-03-30T13:17:51.394431+0000","last_clean":"2024-03-30T13:17:51.394431+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:17:51.394431+0000","last_undegraded":"2024-03-30T13:17:51.394431+0000","last_fullsized":"2024-03-30T13:17:51.394431+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:17:28.727219+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:17:28.727219+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:37.241722+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":106,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.861198+0000","last_change":"2024-03-30T13:17:44.936542+0000","last_active":"2024-03-30T13:17:54.861198+0000","last_peered":"2024-03-30T13:17:54.861198+0000","last_clean":"2024-03-30T13:17:54.861198+0000","last_became_active":"2024-03-30T13:17:44.935396+0000","last_became_peered":"2024-03-30T13:17:44.935396+0000","last_unstale":"2024-03-30T13:17:54.861198+0000","last_undegraded":"2024-03-30T13:17:54.861198+0000","last_fullsized":"2024-03-30T13:17:54.861198+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:16:57.724219+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:16:57.724219+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:23.701163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":645,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.001580+0000","last_change":"2024-03-30T13:17:35.661335+0000","last_active":"2024-03-30T13:17:54.001580+0000","last_peered":"2024-03-30T13:17:54.001580+0000","last_clean":"2024-03-30T13:17:54.001580+0000","last_became_active":"2024-03-30T13:16:51.186350+0000","last_became_peered":"2024-03-30T13:16:51.186350+0000","last_unstale":"2024-03-30T13:17:54.001580+0000","last_undegraded":"2024-03-30T13:17:54.001580+0000","last_fullsized":"2024-03-30T13:17:54.001580+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:17:35.661077+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:17:35.661077+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:03.901602+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1536,"reported_epoch":615,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-30T13:17:51.394280+0000","last_change":"2024-03-30T13:17:42.938450+0000","last_active":"2024-03-30T13:17:51.394280+0000","last_peered":"2024-03-30T13:17:51.394280+0000","last_clean":"2024-03-30T13:17:40.330218+0000","last_became_active":"2024-03-30T13:17:42.921787+0000","last_became_peered":"2024-03-30T13:17:42.921787+0000","last_unstale":"2024-03-30T13:17:51.394280+0000","last_undegraded":"2024-03-30T13:17:51.394280+0000","last_fullsized":"2024-03-30T13:17:51.394280+0000","mapping_epoch":606,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:16:45.572362+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":78420439,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,1],"avail_no_missing":["3(0)","1(2)","5(1)","7(2)"],"object_location_counts":[{"shards":"1(2),3(0),5(1)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1160,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.004281+0000","last_change":"2024-03-30T13:17:36.151905+0000","last_active":"2024-03-30T13:17:54.004281+0000","last_peered":"2024-03-30T13:17:54.004281+0000","last_clean":"2024-03-30T13:17:54.004281+0000","last_became_active":"2024-03-30T13:16:20.315944+0000","last_became_peered":"2024-03-30T13:16:20.315944+0000","last_unstale":"2024-03-30T13:17:54.004281+0000","last_undegraded":"2024-03-30T13:17:54.004281+0000","last_fullsized":"2024-03-30T13:17:54.004281+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:17:36.151638+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_clean_scrub_stamp":"2024-03-30T13:17:36.151638+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:58.250049+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1435,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.860658+0000","last_change":"2024-03-30T13:17:50.532225+0000","last_active":"2024-03-30T13:17:54.860658+0000","last_peered":"2024-03-30T13:17:54.860658+0000","last_clean":"2024-03-30T13:17:54.860658+0000","last_became_active":"2024-03-30T13:16:20.313826+0000","last_became_peered":"2024-03-30T13:16:20.313826+0000","last_unstale":"2024-03-30T13:17:54.860658+0000","last_undegraded":"2024-03-30T13:17:54.860658+0000","last_fullsized":"2024-03-30T13:17:54.860658+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_clean_scrub_stamp":"2024-03-30T13:17:50.531859+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:15.364440+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":38218623,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1505,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.860564+0000","last_change":"2024-03-30T13:17:40.469123+0000","last_active":"2024-03-30T13:17:54.860564+0000","last_peered":"2024-03-30T13:17:54.860564+0000","last_clean":"2024-03-30T13:17:54.860564+0000","last_became_active":"2024-03-30T13:14:38.236424+0000","last_became_peered":"2024-03-30T13:14:38.236424+0000","last_unstale":"2024-03-30T13:17:54.860564+0000","last_undegraded":"2024-03-30T13:17:54.860564+0000","last_fullsized":"2024-03-30T13:17:54.860564+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:17:40.468714+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:17:40.468714+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:48.332593+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":76975004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":264,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-03-30T13:17:51.397126+0000","last_change":"2024-03-30T13:17:40.956701+0000","last_active":"2024-03-30T13:17:51.397126+0000","last_peered":"2024-03-30T13:17:51.397126+0000","last_clean":"2024-03-30T13:17:51.397126+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:17:51.397126+0000","last_undegraded":"2024-03-30T13:17:51.397126+0000","last_fullsized":"2024-03-30T13:17:51.397126+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:17:40.956450+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:17:40.956450+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:59.963567+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":72,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.860814+0000","last_change":"2024-03-30T13:17:44.936616+0000","last_active":"2024-03-30T13:17:54.860814+0000","last_peered":"2024-03-30T13:17:54.860814+0000","last_clean":"2024-03-30T13:17:54.860814+0000","last_became_active":"2024-03-30T13:17:44.935038+0000","last_became_peered":"2024-03-30T13:17:44.935038+0000","last_unstale":"2024-03-30T13:17:54.860814+0000","last_undegraded":"2024-03-30T13:17:54.860814+0000","last_fullsized":"2024-03-30T13:17:54.860814+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:17:19.945177+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:17:19.945177+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:33.414637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":240,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-03-30T13:17:51.397164+0000","last_change":"2024-03-30T13:17:01.851001+0000","last_active":"2024-03-30T13:17:51.397164+0000","last_peered":"2024-03-30T13:17:51.397164+0000","last_clean":"2024-03-30T13:17:51.397164+0000","last_became_active":"2024-03-30T13:14:38.235512+0000","last_became_peered":"2024-03-30T13:14:38.235512+0000","last_unstale":"2024-03-30T13:17:51.397164+0000","last_undegraded":"2024-03-30T13:17:51.397164+0000","last_fullsized":"2024-03-30T13:17:51.397164+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:17:01.850742+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:17:01.850742+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:12.490730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":618,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.001795+0000","last_change":"2024-03-30T13:16:50.991946+0000","last_active":"2024-03-30T13:17:54.001795+0000","last_peered":"2024-03-30T13:17:54.001795+0000","last_clean":"2024-03-30T13:17:54.001795+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:17:54.001795+0000","last_undegraded":"2024-03-30T13:17:54.001795+0000","last_fullsized":"2024-03-30T13:17:54.001795+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:46.315690+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:16:46.315690+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:57.412865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":346774145,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":97374208,"data_stored":97374208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834216,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58776,"kb_used_data":22476,"kb_used_omap":5,"kb_used_meta":36282,"kb_avail":94313064,"statfs":{"total":96636764160,"available":96576577536,"internally_reserved":0,"allocated":23015424,"data_stored":18872783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":37152866},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157481,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64828,"kb_used_data":27184,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94307012,"statfs":{"total":96636764160,"available":96570380288,"internally_reserved":0,"allocated":27836416,"data_stored":23714079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":559,"seq":2400886718485,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56568,"kb_used_data":18924,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94315272,"statfs":{"total":96636764160,"available":96578838528,"internally_reserved":0,"allocated":19378176,"data_stored":15235535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12639,"internal_metadata":38522529},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509712,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49036,"kb_used_data":13692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94322804,"statfs":{"total":96636764160,"available":96586551296,"internally_reserved":0,"allocated":14020608,"data_stored":9877967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375998991,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51500,"kb_used_data":14300,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94320340,"statfs":{"total":96636764160,"available":96584028160,"internally_reserved":0,"allocated":14643200,"data_stored":10500559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509713,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56196,"kb_used_data":20856,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315644,"statfs":{"total":96636764160,"available":96579219456,"internally_reserved":0,"allocated":21356544,"data_stored":17220671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":604,"seq":2594160246789,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54320,"kb_used_data":16676,"kb_used_omap":13,"kb_used_meta":37618,"kb_avail":94317520,"statfs":{"total":96636764160,"available":96581140480,"internally_reserved":0,"allocated":17076224,"data_stored":12940351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14282,"internal_metadata":38520886},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147421,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51528,"kb_used_data":19832,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94320312,"statfs":{"total":96636764160,"available":96583999488,"internally_reserved":0,"allocated":20307968,"data_stored":16185631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7098368,"data_stored":7098368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9543680,"data_stored":9543680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:17:57.400 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:17:57.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:57.517+0000 7f57eb1ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:57.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:57.533+0000 7f57eb1ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:57.706 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:17:57.707 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:17:57.724 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1286,"stamp":"2024-03-30T13:17:57.044007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":346774145,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":49,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":442752,"kb_used_data":153940,"kb_used_omap":59,"kb_used_meta":288644,"kb_avail":754531968,"statfs":{"total":773094113280,"available":772640735232,"internally_reserved":0,"allocated":157634560,"data_stored":124547576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60898,"internal_metadata":295571998},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.051988"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1452,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.861306+0000","last_change":"2024-03-30T13:17:53.569444+0000","last_active":"2024-03-30T13:17:54.861306+0000","last_peered":"2024-03-30T13:17:54.861306+0000","last_clean":"2024-03-30T13:17:54.861306+0000","last_became_active":"2024-03-30T13:16:20.316817+0000","last_became_peered":"2024-03-30T13:16:20.316817+0000","last_unstale":"2024-03-30T13:17:54.861306+0000","last_undegraded":"2024-03-30T13:17:54.861306+0000","last_fullsized":"2024-03-30T13:17:54.861306+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:17:53.569165+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:07.627928+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":32117715,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1314,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.004283+0000","last_change":"2024-03-30T13:17:44.939037+0000","last_active":"2024-03-30T13:17:54.004283+0000","last_peered":"2024-03-30T13:17:54.004283+0000","last_clean":"2024-03-30T13:17:54.004283+0000","last_became_active":"2024-03-30T13:17:44.937731+0000","last_became_peered":"2024-03-30T13:17:44.937731+0000","last_unstale":"2024-03-30T13:17:54.004283+0000","last_undegraded":"2024-03-30T13:17:54.004283+0000","last_fullsized":"2024-03-30T13:17:54.004283+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:17:32.544446+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:17:32.544446+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:33.043891+0000","scrub_duration":331,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":80,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.861339+0000","last_change":"2024-03-30T13:17:44.938352+0000","last_active":"2024-03-30T13:17:54.861339+0000","last_peered":"2024-03-30T13:17:54.861339+0000","last_clean":"2024-03-30T13:17:54.861339+0000","last_became_active":"2024-03-30T13:17:44.937300+0000","last_became_peered":"2024-03-30T13:17:44.937300+0000","last_unstale":"2024-03-30T13:17:54.861339+0000","last_undegraded":"2024-03-30T13:17:54.861339+0000","last_fullsized":"2024-03-30T13:17:54.861339+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:17:13.218985+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:17:13.218985+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:22.528329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":76,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.860970+0000","last_change":"2024-03-30T13:17:46.411658+0000","last_active":"2024-03-30T13:17:54.860970+0000","last_peered":"2024-03-30T13:17:54.860970+0000","last_clean":"2024-03-30T13:17:54.860970+0000","last_became_active":"2024-03-30T13:17:44.934742+0000","last_became_peered":"2024-03-30T13:17:44.934742+0000","last_unstale":"2024-03-30T13:17:54.860970+0000","last_undegraded":"2024-03-30T13:17:54.860970+0000","last_fullsized":"2024-03-30T13:17:54.860970+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:17:46.411360+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:17:46.411360+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:47.789848+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":243,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-03-30T13:17:51.397127+0000","last_change":"2024-03-30T13:16:50.770031+0000","last_active":"2024-03-30T13:17:51.397127+0000","last_peered":"2024-03-30T13:17:51.397127+0000","last_clean":"2024-03-30T13:17:51.397127+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:17:51.397127+0000","last_undegraded":"2024-03-30T13:17:51.397127+0000","last_fullsized":"2024-03-30T13:17:51.397127+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:16:50.769668+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:16:50.769668+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:04.116739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1413,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-03-30T13:17:51.394431+0000","last_change":"2024-03-30T13:17:28.727465+0000","last_active":"2024-03-30T13:17:51.394431+0000","last_peered":"2024-03-30T13:17:51.394431+0000","last_clean":"2024-03-30T13:17:51.394431+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:17:51.394431+0000","last_undegraded":"2024-03-30T13:17:51.394431+0000","last_fullsized":"2024-03-30T13:17:51.394431+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:17:28.727219+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:17:28.727219+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:37.241722+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":106,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.861198+0000","last_change":"2024-03-30T13:17:44.936542+0000","last_active":"2024-03-30T13:17:54.861198+0000","last_peered":"2024-03-30T13:17:54.861198+0000","last_clean":"2024-03-30T13:17:54.861198+0000","last_became_active":"2024-03-30T13:17:44.935396+0000","last_became_peered":"2024-03-30T13:17:44.935396+0000","last_unstale":"2024-03-30T13:17:54.861198+0000","last_undegraded":"2024-03-30T13:17:54.861198+0000","last_fullsized":"2024-03-30T13:17:54.861198+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:16:57.724219+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:16:57.724219+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:23.701163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":645,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.001580+0000","last_change":"2024-03-30T13:17:35.661335+0000","last_active":"2024-03-30T13:17:54.001580+0000","last_peered":"2024-03-30T13:17:54.001580+0000","last_clean":"2024-03-30T13:17:54.001580+0000","last_became_active":"2024-03-30T13:16:51.186350+0000","last_became_peered":"2024-03-30T13:16:51.186350+0000","last_unstale":"2024-03-30T13:17:54.001580+0000","last_undegraded":"2024-03-30T13:17:54.001580+0000","last_fullsized":"2024-03-30T13:17:54.001580+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:17:35.661077+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:17:35.661077+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:03.901602+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1536,"reported_epoch":615,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-30T13:17:51.394280+0000","last_change":"2024-03-30T13:17:42.938450+0000","last_active":"2024-03-30T13:17:51.394280+0000","last_peered":"2024-03-30T13:17:51.394280+0000","last_clean":"2024-03-30T13:17:40.330218+0000","last_became_active":"2024-03-30T13:17:42.921787+0000","last_became_peered":"2024-03-30T13:17:42.921787+0000","last_unstale":"2024-03-30T13:17:51.394280+0000","last_undegraded":"2024-03-30T13:17:51.394280+0000","last_fullsized":"2024-03-30T13:17:51.394280+0000","mapping_epoch":606,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:16:45.572362+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":78420439,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,1],"avail_no_missing":["3(0)","1(2)","5(1)","7(2)"],"object_location_counts":[{"shards":"1(2),3(0),5(1)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1160,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.004281+0000","last_change":"2024-03-30T13:17:36.151905+0000","last_active":"2024-03-30T13:17:54.004281+0000","last_peered":"2024-03-30T13:17:54.004281+0000","last_clean":"2024-03-30T13:17:54.004281+0000","last_became_active":"2024-03-30T13:16:20.315944+0000","last_became_peered":"2024-03-30T13:16:20.315944+0000","last_unstale":"2024-03-30T13:17:54.004281+0000","last_undegraded":"2024-03-30T13:17:54.004281+0000","last_fullsized":"2024-03-30T13:17:54.004281+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:17:36.151638+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_clean_scrub_stamp":"2024-03-30T13:17:36.151638+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:58.250049+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1435,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.860658+0000","last_change":"2024-03-30T13:17:50.532225+0000","last_active":"2024-03-30T13:17:54.860658+0000","last_peered":"2024-03-30T13:17:54.860658+0000","last_clean":"2024-03-30T13:17:54.860658+0000","last_became_active":"2024-03-30T13:16:20.313826+0000","last_became_peered":"2024-03-30T13:16:20.313826+0000","last_unstale":"2024-03-30T13:17:54.860658+0000","last_undegraded":"2024-03-30T13:17:54.860658+0000","last_fullsized":"2024-03-30T13:17:54.860658+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_clean_scrub_stamp":"2024-03-30T13:17:50.531859+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:15.364440+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":38218623,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1505,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.860564+0000","last_change":"2024-03-30T13:17:40.469123+0000","last_active":"2024-03-30T13:17:54.860564+0000","last_peered":"2024-03-30T13:17:54.860564+0000","last_clean":"2024-03-30T13:17:54.860564+0000","last_became_active":"2024-03-30T13:14:38.236424+0000","last_became_peered":"2024-03-30T13:14:38.236424+0000","last_unstale":"2024-03-30T13:17:54.860564+0000","last_undegraded":"2024-03-30T13:17:54.860564+0000","last_fullsized":"2024-03-30T13:17:54.860564+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:17:40.468714+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:17:40.468714+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:48.332593+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":76975004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":264,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-03-30T13:17:51.397126+0000","last_change":"2024-03-30T13:17:40.956701+0000","last_active":"2024-03-30T13:17:51.397126+0000","last_peered":"2024-03-30T13:17:51.397126+0000","last_clean":"2024-03-30T13:17:51.397126+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:17:51.397126+0000","last_undegraded":"2024-03-30T13:17:51.397126+0000","last_fullsized":"2024-03-30T13:17:51.397126+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:17:40.956450+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:17:40.956450+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:59.963567+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":72,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.860814+0000","last_change":"2024-03-30T13:17:44.936616+0000","last_active":"2024-03-30T13:17:54.860814+0000","last_peered":"2024-03-30T13:17:54.860814+0000","last_clean":"2024-03-30T13:17:54.860814+0000","last_became_active":"2024-03-30T13:17:44.935038+0000","last_became_peered":"2024-03-30T13:17:44.935038+0000","last_unstale":"2024-03-30T13:17:54.860814+0000","last_undegraded":"2024-03-30T13:17:54.860814+0000","last_fullsized":"2024-03-30T13:17:54.860814+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:17:19.945177+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:17:19.945177+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:33.414637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":240,"reported_epoch":615,"state":"active+clean","last_fresh":"2024-03-30T13:17:51.397164+0000","last_change":"2024-03-30T13:17:01.851001+0000","last_active":"2024-03-30T13:17:51.397164+0000","last_peered":"2024-03-30T13:17:51.397164+0000","last_clean":"2024-03-30T13:17:51.397164+0000","last_became_active":"2024-03-30T13:14:38.235512+0000","last_became_peered":"2024-03-30T13:14:38.235512+0000","last_unstale":"2024-03-30T13:17:51.397164+0000","last_undegraded":"2024-03-30T13:17:51.397164+0000","last_fullsized":"2024-03-30T13:17:51.397164+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:17:01.850742+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:17:01.850742+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:12.490730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":618,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.001795+0000","last_change":"2024-03-30T13:16:50.991946+0000","last_active":"2024-03-30T13:17:54.001795+0000","last_peered":"2024-03-30T13:17:54.001795+0000","last_clean":"2024-03-30T13:17:54.001795+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:17:54.001795+0000","last_undegraded":"2024-03-30T13:17:54.001795+0000","last_fullsized":"2024-03-30T13:17:54.001795+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:46.315690+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:16:46.315690+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:57.412865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":346774145,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":97374208,"data_stored":97374208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834216,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58776,"kb_used_data":22476,"kb_used_omap":5,"kb_used_meta":36282,"kb_avail":94313064,"statfs":{"total":96636764160,"available":96576577536,"internally_reserved":0,"allocated":23015424,"data_stored":18872783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":37152866},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157481,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64828,"kb_used_data":27184,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94307012,"statfs":{"total":96636764160,"available":96570380288,"internally_reserved":0,"allocated":27836416,"data_stored":23714079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":559,"seq":2400886718485,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56568,"kb_used_data":18924,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94315272,"statfs":{"total":96636764160,"available":96578838528,"internally_reserved":0,"allocated":19378176,"data_stored":15235535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12639,"internal_metadata":38522529},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509712,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49036,"kb_used_data":13692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94322804,"statfs":{"total":96636764160,"available":96586551296,"internally_reserved":0,"allocated":14020608,"data_stored":9877967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375998991,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51500,"kb_used_data":14300,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94320340,"statfs":{"total":96636764160,"available":96584028160,"internally_reserved":0,"allocated":14643200,"data_stored":10500559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509713,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56196,"kb_used_data":20856,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315644,"statfs":{"total":96636764160,"available":96579219456,"internally_reserved":0,"allocated":21356544,"data_stored":17220671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":604,"seq":2594160246789,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54320,"kb_used_data":16676,"kb_used_omap":13,"kb_used_meta":37618,"kb_avail":94317520,"statfs":{"total":96636764160,"available":96581140480,"internally_reserved":0,"allocated":17076224,"data_stored":12940351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14282,"internal_metadata":38520886},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147421,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51528,"kb_used_data":19832,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94320312,"statfs":{"total":96636764160,"available":96583999488,"internally_reserved":0,"allocated":20307968,"data_stored":16185631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7098368,"data_stored":7098368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9543680,"data_stored":9543680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:17:57.725 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:17:57.726 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:17:57.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:57.837+0000 7f5768bfd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:57.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:57.849+0000 7f5768bfd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:58.023 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:17:58.023 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":621,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:17:57.029663+0000","last_up_change":"2024-03-30T13:17:40.319102+0000","last_in_change":"2024-03-30T13:17:41.326209+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":31,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":598,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":461,"last_clean_end":575,"up_from":581,"up_thru":607,"down_at":576,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6818","nonce":4030404750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6821","nonce":4030404750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6828","nonce":4030404750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6825","nonce":4030404750}]},"public_addr":"172.21.15.67:6818/4030404750","cluster_addr":"172.21.15.67:6821/4030404750","heartbeat_back_addr":"172.21.15.67:6828/4030404750","heartbeat_front_addr":"172.21.15.67:6825/4030404750","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":607,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":388,"up_from":393,"up_thru":606,"down_at":389,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6801","nonce":3837194718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6803","nonce":3837194718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6807","nonce":3837194718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6805","nonce":3837194718}]},"public_addr":"172.21.15.67:6801/3837194718","cluster_addr":"172.21.15.67:6803/3837194718","heartbeat_back_addr":"172.21.15.67:6807/3837194718","heartbeat_front_addr":"172.21.15.67:6805/3837194718","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":536,"last_clean_end":552,"up_from":559,"up_thru":608,"down_at":553,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3095958959},{"type":"v1","addr":"172.21.15.82:6803","nonce":3095958959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":3095958959},{"type":"v1","addr":"172.21.15.82:6807","nonce":3095958959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":3095958959},{"type":"v1","addr":"172.21.15.82:6815","nonce":3095958959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":3095958959},{"type":"v1","addr":"172.21.15.82:6811","nonce":3095958959}]},"public_addr":"172.21.15.82:6803/3095958959","cluster_addr":"172.21.15.82:6807/3095958959","heartbeat_back_addr":"172.21.15.82:6815/3095958959","heartbeat_front_addr":"172.21.15.82:6811/3095958959","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":488,"last_clean_end":506,"up_from":515,"up_thru":607,"down_at":507,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6817","nonce":292505453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6819","nonce":292505453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6823","nonce":292505453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6821","nonce":292505453}]},"public_addr":"172.21.15.82:6817/292505453","cluster_addr":"172.21.15.82:6819/292505453","heartbeat_back_addr":"172.21.15.82:6823/292505453","heartbeat_front_addr":"172.21.15.82:6821/292505453","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":227,"last_clean_end":363,"up_from":372,"up_thru":576,"down_at":364,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6801","nonce":189641108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6806","nonce":189641108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6814","nonce":189641108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":189641108},{"type":"v1","addr":"172.21.15.82:6810","nonce":189641108}]},"public_addr":"172.21.15.82:6801/189641108","cluster_addr":"172.21.15.82:6806/189641108","heartbeat_back_addr":"172.21.15.82:6814/189641108","heartbeat_front_addr":"172.21.15.82:6810/189641108","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":419,"last_clean_end":597,"up_from":604,"up_thru":608,"down_at":598,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3016824960},{"type":"v1","addr":"172.21.15.82:6825","nonce":3016824960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3016824960},{"type":"v1","addr":"172.21.15.82:6827","nonce":3016824960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3016824960},{"type":"v1","addr":"172.21.15.82:6831","nonce":3016824960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3016824960},{"type":"v1","addr":"172.21.15.82:6829","nonce":3016824960}]},"public_addr":"172.21.15.82:6825/3016824960","cluster_addr":"172.21.15.82:6827/3016824960","heartbeat_back_addr":"172.21.15.82:6831/3016824960","heartbeat_front_addr":"172.21.15.82:6829/3016824960","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:17:16.234435+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:17:39.736123+0000","dead_epoch":576},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:16:56.914201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:17:16.598887+0000","dead_epoch":389},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:16:59.641873+0000","dead_epoch":553},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:17:18.113711+0000","dead_epoch":507},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:16:53.943474+0000","dead_epoch":364},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:16:56.924723+0000","dead_epoch":598}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"1.4","osds":[3,5,1]}],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:17:58.042 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:17:58.042 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:17:58.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:58.157+0000 7feaa0406640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:58.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:58.177+0000 7feaa0406640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:58.351 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:17:58.370 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:17:58.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:58.481+0000 7f4f1922d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:58.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:58.497+0000 7f4f1922d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:58.669 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:17:58.670 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:17:58.687 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:17:58.687 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:17:58.687 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:17:58.688 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:17:58.688 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 6, 3, 5, 4, 1, 7] 2024-03-30T13:17:58.688 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:17:58.688 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:17:58.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:58.793+0000 7f7a6ac74640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:58.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:58.809+0000 7f7a6ac74640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:58.984 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:17:58.985 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:17:59.001 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1287,"stamp":"2024-03-30T13:17:58.171543+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":346774145,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":49,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":442880,"kb_used_data":154060,"kb_used_omap":59,"kb_used_meta":288644,"kb_avail":754531840,"statfs":{"total":773094113280,"available":772640604160,"internally_reserved":0,"allocated":157757440,"data_stored":124602776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":60898,"internal_metadata":295571998},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.169495"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1452,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.861306+0000","last_change":"2024-03-30T13:17:53.569444+0000","last_active":"2024-03-30T13:17:54.861306+0000","last_peered":"2024-03-30T13:17:54.861306+0000","last_clean":"2024-03-30T13:17:54.861306+0000","last_became_active":"2024-03-30T13:16:20.316817+0000","last_became_peered":"2024-03-30T13:16:20.316817+0000","last_unstale":"2024-03-30T13:17:54.861306+0000","last_undegraded":"2024-03-30T13:17:54.861306+0000","last_fullsized":"2024-03-30T13:17:54.861306+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:17:53.569165+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:07.627928+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":32117715,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1314,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.004283+0000","last_change":"2024-03-30T13:17:44.939037+0000","last_active":"2024-03-30T13:17:54.004283+0000","last_peered":"2024-03-30T13:17:54.004283+0000","last_clean":"2024-03-30T13:17:54.004283+0000","last_became_active":"2024-03-30T13:17:44.937731+0000","last_became_peered":"2024-03-30T13:17:44.937731+0000","last_unstale":"2024-03-30T13:17:54.004283+0000","last_undegraded":"2024-03-30T13:17:54.004283+0000","last_fullsized":"2024-03-30T13:17:54.004283+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:17:32.544446+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:17:32.544446+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:33.043891+0000","scrub_duration":331,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":80,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.861339+0000","last_change":"2024-03-30T13:17:44.938352+0000","last_active":"2024-03-30T13:17:54.861339+0000","last_peered":"2024-03-30T13:17:54.861339+0000","last_clean":"2024-03-30T13:17:54.861339+0000","last_became_active":"2024-03-30T13:17:44.937300+0000","last_became_peered":"2024-03-30T13:17:44.937300+0000","last_unstale":"2024-03-30T13:17:54.861339+0000","last_undegraded":"2024-03-30T13:17:54.861339+0000","last_fullsized":"2024-03-30T13:17:54.861339+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:17:13.218985+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:17:13.218985+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:22.528329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":76,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.860970+0000","last_change":"2024-03-30T13:17:46.411658+0000","last_active":"2024-03-30T13:17:54.860970+0000","last_peered":"2024-03-30T13:17:54.860970+0000","last_clean":"2024-03-30T13:17:54.860970+0000","last_became_active":"2024-03-30T13:17:44.934742+0000","last_became_peered":"2024-03-30T13:17:44.934742+0000","last_unstale":"2024-03-30T13:17:54.860970+0000","last_undegraded":"2024-03-30T13:17:54.860970+0000","last_fullsized":"2024-03-30T13:17:54.860970+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:17:46.411360+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:17:46.411360+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:47.789848+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":252,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-03-30T13:17:56.697170+0000","last_change":"2024-03-30T13:16:50.770031+0000","last_active":"2024-03-30T13:17:56.697170+0000","last_peered":"2024-03-30T13:17:56.697170+0000","last_clean":"2024-03-30T13:17:56.697170+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:17:56.697170+0000","last_undegraded":"2024-03-30T13:17:56.697170+0000","last_fullsized":"2024-03-30T13:17:56.697170+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:16:50.769668+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:16:50.769668+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:04.116739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1422,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-03-30T13:17:56.696744+0000","last_change":"2024-03-30T13:17:28.727465+0000","last_active":"2024-03-30T13:17:56.696744+0000","last_peered":"2024-03-30T13:17:56.696744+0000","last_clean":"2024-03-30T13:17:56.696744+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:17:56.696744+0000","last_undegraded":"2024-03-30T13:17:56.696744+0000","last_fullsized":"2024-03-30T13:17:56.696744+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:17:28.727219+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:17:28.727219+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:37.241722+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":106,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.861198+0000","last_change":"2024-03-30T13:17:44.936542+0000","last_active":"2024-03-30T13:17:54.861198+0000","last_peered":"2024-03-30T13:17:54.861198+0000","last_clean":"2024-03-30T13:17:54.861198+0000","last_became_active":"2024-03-30T13:17:44.935396+0000","last_became_peered":"2024-03-30T13:17:44.935396+0000","last_unstale":"2024-03-30T13:17:54.861198+0000","last_undegraded":"2024-03-30T13:17:54.861198+0000","last_fullsized":"2024-03-30T13:17:54.861198+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:16:57.724219+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:16:57.724219+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:23.701163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":645,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.001580+0000","last_change":"2024-03-30T13:17:35.661335+0000","last_active":"2024-03-30T13:17:54.001580+0000","last_peered":"2024-03-30T13:17:54.001580+0000","last_clean":"2024-03-30T13:17:54.001580+0000","last_became_active":"2024-03-30T13:16:51.186350+0000","last_became_peered":"2024-03-30T13:16:51.186350+0000","last_unstale":"2024-03-30T13:17:54.001580+0000","last_undegraded":"2024-03-30T13:17:54.001580+0000","last_fullsized":"2024-03-30T13:17:54.001580+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:17:35.661077+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:17:35.661077+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:03.901602+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1545,"reported_epoch":620,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-30T13:17:56.696728+0000","last_change":"2024-03-30T13:17:42.938450+0000","last_active":"2024-03-30T13:17:56.696728+0000","last_peered":"2024-03-30T13:17:56.696728+0000","last_clean":"2024-03-30T13:17:40.330218+0000","last_became_active":"2024-03-30T13:17:42.921787+0000","last_became_peered":"2024-03-30T13:17:42.921787+0000","last_unstale":"2024-03-30T13:17:56.696728+0000","last_undegraded":"2024-03-30T13:17:56.696728+0000","last_fullsized":"2024-03-30T13:17:56.696728+0000","mapping_epoch":606,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:16:45.572362+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":78420439,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,1],"avail_no_missing":["3(0)","1(2)","5(1)","7(2)"],"object_location_counts":[{"shards":"1(2),3(0),5(1)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1160,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.004281+0000","last_change":"2024-03-30T13:17:36.151905+0000","last_active":"2024-03-30T13:17:54.004281+0000","last_peered":"2024-03-30T13:17:54.004281+0000","last_clean":"2024-03-30T13:17:54.004281+0000","last_became_active":"2024-03-30T13:16:20.315944+0000","last_became_peered":"2024-03-30T13:16:20.315944+0000","last_unstale":"2024-03-30T13:17:54.004281+0000","last_undegraded":"2024-03-30T13:17:54.004281+0000","last_fullsized":"2024-03-30T13:17:54.004281+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:17:36.151638+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_clean_scrub_stamp":"2024-03-30T13:17:36.151638+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:58.250049+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1435,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.860658+0000","last_change":"2024-03-30T13:17:50.532225+0000","last_active":"2024-03-30T13:17:54.860658+0000","last_peered":"2024-03-30T13:17:54.860658+0000","last_clean":"2024-03-30T13:17:54.860658+0000","last_became_active":"2024-03-30T13:16:20.313826+0000","last_became_peered":"2024-03-30T13:16:20.313826+0000","last_unstale":"2024-03-30T13:17:54.860658+0000","last_undegraded":"2024-03-30T13:17:54.860658+0000","last_fullsized":"2024-03-30T13:17:54.860658+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_clean_scrub_stamp":"2024-03-30T13:17:50.531859+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:15.364440+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":38218623,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1505,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.860564+0000","last_change":"2024-03-30T13:17:40.469123+0000","last_active":"2024-03-30T13:17:54.860564+0000","last_peered":"2024-03-30T13:17:54.860564+0000","last_clean":"2024-03-30T13:17:54.860564+0000","last_became_active":"2024-03-30T13:14:38.236424+0000","last_became_peered":"2024-03-30T13:14:38.236424+0000","last_unstale":"2024-03-30T13:17:54.860564+0000","last_undegraded":"2024-03-30T13:17:54.860564+0000","last_fullsized":"2024-03-30T13:17:54.860564+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:17:40.468714+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:17:40.468714+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:48.332593+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":76975004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":273,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-03-30T13:17:56.697161+0000","last_change":"2024-03-30T13:17:40.956701+0000","last_active":"2024-03-30T13:17:56.697161+0000","last_peered":"2024-03-30T13:17:56.697161+0000","last_clean":"2024-03-30T13:17:56.697161+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:17:56.697161+0000","last_undegraded":"2024-03-30T13:17:56.697161+0000","last_fullsized":"2024-03-30T13:17:56.697161+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:17:40.956450+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:17:40.956450+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:59.963567+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":72,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.860814+0000","last_change":"2024-03-30T13:17:44.936616+0000","last_active":"2024-03-30T13:17:54.860814+0000","last_peered":"2024-03-30T13:17:54.860814+0000","last_clean":"2024-03-30T13:17:54.860814+0000","last_became_active":"2024-03-30T13:17:44.935038+0000","last_became_peered":"2024-03-30T13:17:44.935038+0000","last_unstale":"2024-03-30T13:17:54.860814+0000","last_undegraded":"2024-03-30T13:17:54.860814+0000","last_fullsized":"2024-03-30T13:17:54.860814+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:17:19.945177+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:17:19.945177+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:33.414637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":249,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-03-30T13:17:56.697191+0000","last_change":"2024-03-30T13:17:01.851001+0000","last_active":"2024-03-30T13:17:56.697191+0000","last_peered":"2024-03-30T13:17:56.697191+0000","last_clean":"2024-03-30T13:17:56.697191+0000","last_became_active":"2024-03-30T13:14:38.235512+0000","last_became_peered":"2024-03-30T13:14:38.235512+0000","last_unstale":"2024-03-30T13:17:56.697191+0000","last_undegraded":"2024-03-30T13:17:56.697191+0000","last_fullsized":"2024-03-30T13:17:56.697191+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:17:01.850742+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:17:01.850742+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:12.490730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":618,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.001795+0000","last_change":"2024-03-30T13:16:50.991946+0000","last_active":"2024-03-30T13:17:54.001795+0000","last_peered":"2024-03-30T13:17:54.001795+0000","last_clean":"2024-03-30T13:17:54.001795+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:17:54.001795+0000","last_undegraded":"2024-03-30T13:17:54.001795+0000","last_fullsized":"2024-03-30T13:17:54.001795+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:46.315690+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:01:27.958371+0000","last_clean_scrub_stamp":"2024-03-30T13:16:46.315690+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:17:57.412865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":346774145,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":97374208,"data_stored":97374208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":372,"seq":1597727834216,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58776,"kb_used_data":22476,"kb_used_omap":5,"kb_used_meta":36282,"kb_avail":94313064,"statfs":{"total":96636764160,"available":96576577536,"internally_reserved":0,"allocated":23015424,"data_stored":18872783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":37152866},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157482,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64896,"kb_used_data":27244,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94306944,"statfs":{"total":96636764160,"available":96570310656,"internally_reserved":0,"allocated":27897856,"data_stored":23741679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":559,"seq":2400886718485,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56568,"kb_used_data":18924,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94315272,"statfs":{"total":96636764160,"available":96578838528,"internally_reserved":0,"allocated":19378176,"data_stored":15235535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12639,"internal_metadata":38522529},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509712,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49036,"kb_used_data":13692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94322804,"statfs":{"total":96636764160,"available":96586551296,"internally_reserved":0,"allocated":14020608,"data_stored":9877967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375998991,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51500,"kb_used_data":14300,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94320340,"statfs":{"total":96636764160,"available":96584028160,"internally_reserved":0,"allocated":14643200,"data_stored":10500559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509713,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56196,"kb_used_data":20856,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315644,"statfs":{"total":96636764160,"available":96579219456,"internally_reserved":0,"allocated":21356544,"data_stored":17220671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":604,"seq":2594160246789,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54320,"kb_used_data":16676,"kb_used_omap":13,"kb_used_meta":37618,"kb_avail":94317520,"statfs":{"total":96636764160,"available":96581140480,"internally_reserved":0,"allocated":17076224,"data_stored":12940351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14282,"internal_metadata":38520886},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147422,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51588,"kb_used_data":19892,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94320252,"statfs":{"total":96636764160,"available":96583938048,"internally_reserved":0,"allocated":20369408,"data_stored":16213231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7098368,"data_stored":7098368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9543680,"data_stored":9543680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:17:59.002 INFO:tasks.thrashosds.thrasher:Choosing PG 1.6 with acting set [6, 0, 4] 2024-03-30T13:17:59.002 INFO:tasks.thrashosds.thrasher:Killing osd 6, live_osds are [0, 2, 6, 3, 5, 4, 1, 7] 2024-03-30T13:17:59.002 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-30T13:17:59.002 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:17:59.058 INFO:tasks.ceph.osd.6:Stopped 2024-03-30T13:17:59.059 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 2, 6, 3, 5, 4, 1, 7] 2024-03-30T13:17:59.059 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 6 2024-03-30T13:17:59.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:59.173+0000 7f2cbf359640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:59.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:17:59.185+0000 7f2cbf359640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:17:59.362 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:17:59.361+0000 7fbb38f08640 -1 mon.a@0(leader).osd e621 definitely_dead 0 2024-03-30T13:18:00.054 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.6. 2024-03-30T13:18:00.082 INFO:tasks.thrashosds.thrasher:dead_osds=[6], live_osds=[0, 2, 3, 5, 4, 1, 7] 2024-03-30T13:18:00.082 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:18:00.082 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:18:00.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:00.197+0000 7f3f259e9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:00.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:00.213+0000 7f3f259e9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:00.385 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:18:00.385 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:18:00.403 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1289,"stamp":"2024-03-30T13:18:00.172176+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":346774145,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":384256,"kb_used_data":131740,"kb_used_omap":53,"kb_used_meta":252362,"kb_avail":660218624,"statfs":{"total":676457349120,"available":676063870976,"internally_reserved":0,"allocated":134901760,"data_stored":105801753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":54852,"internal_metadata":258419132},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.152688"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1452,"reported_epoch":618,"state":"stale+active+clean","last_fresh":"2024-03-30T13:17:54.861306+0000","last_change":"2024-03-30T13:17:53.569444+0000","last_active":"2024-03-30T13:17:54.861306+0000","last_peered":"2024-03-30T13:17:54.861306+0000","last_clean":"2024-03-30T13:17:54.861306+0000","last_became_active":"2024-03-30T13:16:20.316817+0000","last_became_peered":"2024-03-30T13:16:20.316817+0000","last_unstale":"2024-03-30T13:18:00.063565+0000","last_undegraded":"2024-03-30T13:17:54.861306+0000","last_fullsized":"2024-03-30T13:17:54.861306+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:17:53.569165+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:07.627928+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":32117715,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1320,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-03-30T13:17:57.046250+0000","last_change":"2024-03-30T13:17:44.939037+0000","last_active":"2024-03-30T13:17:57.046250+0000","last_peered":"2024-03-30T13:17:57.046250+0000","last_clean":"2024-03-30T13:17:57.046250+0000","last_became_active":"2024-03-30T13:17:44.937731+0000","last_became_peered":"2024-03-30T13:17:44.937731+0000","last_unstale":"2024-03-30T13:17:57.046250+0000","last_undegraded":"2024-03-30T13:17:57.046250+0000","last_fullsized":"2024-03-30T13:17:57.046250+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:17:32.544446+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:17:32.544446+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:33.043891+0000","scrub_duration":331,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":80,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.861339+0000","last_change":"2024-03-30T13:17:44.938352+0000","last_active":"2024-03-30T13:17:54.861339+0000","last_peered":"2024-03-30T13:17:54.861339+0000","last_clean":"2024-03-30T13:17:54.861339+0000","last_became_active":"2024-03-30T13:17:44.937300+0000","last_became_peered":"2024-03-30T13:17:44.937300+0000","last_unstale":"2024-03-30T13:17:54.861339+0000","last_undegraded":"2024-03-30T13:17:54.861339+0000","last_fullsized":"2024-03-30T13:17:54.861339+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:17:13.218985+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:17:13.218985+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:22.528329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":76,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.860970+0000","last_change":"2024-03-30T13:17:46.411658+0000","last_active":"2024-03-30T13:17:54.860970+0000","last_peered":"2024-03-30T13:17:54.860970+0000","last_clean":"2024-03-30T13:17:54.860970+0000","last_became_active":"2024-03-30T13:17:44.934742+0000","last_became_peered":"2024-03-30T13:17:44.934742+0000","last_unstale":"2024-03-30T13:17:54.860970+0000","last_undegraded":"2024-03-30T13:17:54.860970+0000","last_fullsized":"2024-03-30T13:17:54.860970+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:17:46.411360+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:17:46.411360+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:47.789848+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":252,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-03-30T13:17:56.697170+0000","last_change":"2024-03-30T13:16:50.770031+0000","last_active":"2024-03-30T13:17:56.697170+0000","last_peered":"2024-03-30T13:17:56.697170+0000","last_clean":"2024-03-30T13:17:56.697170+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:17:56.697170+0000","last_undegraded":"2024-03-30T13:17:56.697170+0000","last_fullsized":"2024-03-30T13:17:56.697170+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:16:50.769668+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:16:50.769668+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:04.116739+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1422,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-03-30T13:17:56.696744+0000","last_change":"2024-03-30T13:17:28.727465+0000","last_active":"2024-03-30T13:17:56.696744+0000","last_peered":"2024-03-30T13:17:56.696744+0000","last_clean":"2024-03-30T13:17:56.696744+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:17:56.696744+0000","last_undegraded":"2024-03-30T13:17:56.696744+0000","last_fullsized":"2024-03-30T13:17:56.696744+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:17:28.727219+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:17:28.727219+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:37.241722+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":106,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.861198+0000","last_change":"2024-03-30T13:17:44.936542+0000","last_active":"2024-03-30T13:17:54.861198+0000","last_peered":"2024-03-30T13:17:54.861198+0000","last_clean":"2024-03-30T13:17:54.861198+0000","last_became_active":"2024-03-30T13:17:44.935396+0000","last_became_peered":"2024-03-30T13:17:44.935396+0000","last_unstale":"2024-03-30T13:17:54.861198+0000","last_undegraded":"2024-03-30T13:17:54.861198+0000","last_fullsized":"2024-03-30T13:17:54.861198+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:16:57.724219+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:16:57.724219+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:23.701163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":651,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-03-30T13:17:57.713192+0000","last_change":"2024-03-30T13:17:35.661335+0000","last_active":"2024-03-30T13:17:57.713192+0000","last_peered":"2024-03-30T13:17:57.713192+0000","last_clean":"2024-03-30T13:17:57.713192+0000","last_became_active":"2024-03-30T13:16:51.186350+0000","last_became_peered":"2024-03-30T13:16:51.186350+0000","last_unstale":"2024-03-30T13:17:57.713192+0000","last_undegraded":"2024-03-30T13:17:57.713192+0000","last_fullsized":"2024-03-30T13:17:57.713192+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:17:35.661077+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:17:35.661077+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:03.901602+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1545,"reported_epoch":620,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-30T13:17:56.696728+0000","last_change":"2024-03-30T13:17:42.938450+0000","last_active":"2024-03-30T13:17:56.696728+0000","last_peered":"2024-03-30T13:17:56.696728+0000","last_clean":"2024-03-30T13:17:40.330218+0000","last_became_active":"2024-03-30T13:17:42.921787+0000","last_became_peered":"2024-03-30T13:17:42.921787+0000","last_unstale":"2024-03-30T13:17:56.696728+0000","last_undegraded":"2024-03-30T13:17:56.696728+0000","last_fullsized":"2024-03-30T13:17:56.696728+0000","mapping_epoch":606,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:16:45.572362+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":78420439,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,1],"avail_no_missing":["3(0)","1(2)","5(1)","7(2)"],"object_location_counts":[{"shards":"1(2),3(0),5(1)","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1166,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-03-30T13:17:57.045998+0000","last_change":"2024-03-30T13:17:36.151905+0000","last_active":"2024-03-30T13:17:57.045998+0000","last_peered":"2024-03-30T13:17:57.045998+0000","last_clean":"2024-03-30T13:17:57.045998+0000","last_became_active":"2024-03-30T13:16:20.315944+0000","last_became_peered":"2024-03-30T13:16:20.315944+0000","last_unstale":"2024-03-30T13:17:57.045998+0000","last_undegraded":"2024-03-30T13:17:57.045998+0000","last_fullsized":"2024-03-30T13:17:57.045998+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:17:36.151638+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_clean_scrub_stamp":"2024-03-30T13:17:36.151638+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:58.250049+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1435,"reported_epoch":618,"state":"stale+active+clean","last_fresh":"2024-03-30T13:17:54.860658+0000","last_change":"2024-03-30T13:17:50.532225+0000","last_active":"2024-03-30T13:17:54.860658+0000","last_peered":"2024-03-30T13:17:54.860658+0000","last_clean":"2024-03-30T13:17:54.860658+0000","last_became_active":"2024-03-30T13:16:20.313826+0000","last_became_peered":"2024-03-30T13:16:20.313826+0000","last_unstale":"2024-03-30T13:18:00.063570+0000","last_undegraded":"2024-03-30T13:17:54.860658+0000","last_fullsized":"2024-03-30T13:17:54.860658+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_clean_scrub_stamp":"2024-03-30T13:17:50.531859+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:15.364440+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":38218623,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1505,"reported_epoch":618,"state":"stale+active+clean","last_fresh":"2024-03-30T13:17:54.860564+0000","last_change":"2024-03-30T13:17:40.469123+0000","last_active":"2024-03-30T13:17:54.860564+0000","last_peered":"2024-03-30T13:17:54.860564+0000","last_clean":"2024-03-30T13:17:54.860564+0000","last_became_active":"2024-03-30T13:14:38.236424+0000","last_became_peered":"2024-03-30T13:14:38.236424+0000","last_unstale":"2024-03-30T13:18:00.063573+0000","last_undegraded":"2024-03-30T13:17:54.860564+0000","last_fullsized":"2024-03-30T13:17:54.860564+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:17:40.468714+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:17:40.468714+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:48.332593+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":76975004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":273,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-03-30T13:17:56.697161+0000","last_change":"2024-03-30T13:17:40.956701+0000","last_active":"2024-03-30T13:17:56.697161+0000","last_peered":"2024-03-30T13:17:56.697161+0000","last_clean":"2024-03-30T13:17:56.697161+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:17:56.697161+0000","last_undegraded":"2024-03-30T13:17:56.697161+0000","last_fullsized":"2024-03-30T13:17:56.697161+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:17:40.956450+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:17:40.956450+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:59.963567+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":72,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-03-30T13:17:54.860814+0000","last_change":"2024-03-30T13:17:44.936616+0000","last_active":"2024-03-30T13:17:54.860814+0000","last_peered":"2024-03-30T13:17:54.860814+0000","last_clean":"2024-03-30T13:17:54.860814+0000","last_became_active":"2024-03-30T13:17:44.935038+0000","last_became_peered":"2024-03-30T13:17:44.935038+0000","last_unstale":"2024-03-30T13:17:54.860814+0000","last_undegraded":"2024-03-30T13:17:54.860814+0000","last_fullsized":"2024-03-30T13:17:54.860814+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:17:19.945177+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:17:19.945177+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:33.414637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":249,"reported_epoch":620,"state":"active+clean","last_fresh":"2024-03-30T13:17:56.697191+0000","last_change":"2024-03-30T13:17:01.851001+0000","last_active":"2024-03-30T13:17:56.697191+0000","last_peered":"2024-03-30T13:17:56.697191+0000","last_clean":"2024-03-30T13:17:56.697191+0000","last_became_active":"2024-03-30T13:14:38.235512+0000","last_became_peered":"2024-03-30T13:14:38.235512+0000","last_unstale":"2024-03-30T13:17:56.697191+0000","last_undegraded":"2024-03-30T13:17:56.697191+0000","last_fullsized":"2024-03-30T13:17:56.697191+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:17:01.850742+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:17:01.850742+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:12.490730+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":632,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-03-30T13:17:58.851082+0000","last_change":"2024-03-30T13:17:58.851110+0000","last_active":"2024-03-30T13:17:58.851082+0000","last_peered":"2024-03-30T13:17:58.851082+0000","last_clean":"2024-03-30T13:17:58.851082+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:17:58.851082+0000","last_undegraded":"2024-03-30T13:17:58.851082+0000","last_fullsized":"2024-03-30T13:17:58.851082+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:17:58.850996+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:15.607294+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":346774145,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":97374208,"data_stored":97374208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157482,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64896,"kb_used_data":27244,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94306944,"statfs":{"total":96636764160,"available":96570310656,"internally_reserved":0,"allocated":27897856,"data_stored":23741679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":559,"seq":2400886718486,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56604,"kb_used_data":18960,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94315236,"statfs":{"total":96636764160,"available":96578801664,"internally_reserved":0,"allocated":19415040,"data_stored":15252095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12639,"internal_metadata":38522529},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509713,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49072,"kb_used_data":13728,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94322768,"statfs":{"total":96636764160,"available":96586514432,"internally_reserved":0,"allocated":14057472,"data_stored":9894527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375998992,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51528,"kb_used_data":14336,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94320312,"statfs":{"total":96636764160,"available":96583999488,"internally_reserved":0,"allocated":14680064,"data_stored":10517119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509714,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56248,"kb_used_data":20904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315592,"statfs":{"total":96636764160,"available":96579166208,"internally_reserved":0,"allocated":21405696,"data_stored":17242751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":604,"seq":2594160246789,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54320,"kb_used_data":16676,"kb_used_omap":13,"kb_used_meta":37618,"kb_avail":94317520,"statfs":{"total":96636764160,"available":96581140480,"internally_reserved":0,"allocated":17076224,"data_stored":12940351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14282,"internal_metadata":38520886},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147422,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51588,"kb_used_data":19892,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94320252,"statfs":{"total":96636764160,"available":96583938048,"internally_reserved":0,"allocated":20369408,"data_stored":16213231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":7098368,"data_stored":7098368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":9543680,"data_stored":9543680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:18:00.404 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:18:25.405 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:18:25.406 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:18:25.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:25.532+0000 7fbe98f34640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:25.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:25.544+0000 7fbe98f34640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:25.720 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:18:25.720 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:18:25.737 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1305,"stamp":"2024-03-30T13:18:24.179493+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":262168862,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":400492,"kb_used_data":147948,"kb_used_omap":53,"kb_used_meta":252362,"kb_avail":660202388,"statfs":{"total":676457349120,"available":676047245312,"internally_reserved":0,"allocated":151498752,"data_stored":122123014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":54852,"internal_metadata":258419132},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2789929,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.914454"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":23,"reported_epoch":626,"state":"active+clean","last_fresh":"2024-03-30T13:18:15.277612+0000","last_change":"2024-03-30T13:18:01.139419+0000","last_active":"2024-03-30T13:18:15.277612+0000","last_peered":"2024-03-30T13:18:15.277612+0000","last_clean":"2024-03-30T13:18:15.277612+0000","last_became_active":"2024-03-30T13:18:01.083700+0000","last_became_peered":"2024-03-30T13:18:01.083700+0000","last_unstale":"2024-03-30T13:18:15.277612+0000","last_undegraded":"2024-03-30T13:18:15.277612+0000","last_fullsized":"2024-03-30T13:18:15.277612+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:17:53.569165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:09.400139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1330,"reported_epoch":626,"state":"active+clean","last_fresh":"2024-03-30T13:18:15.276358+0000","last_change":"2024-03-30T13:17:44.939037+0000","last_active":"2024-03-30T13:18:15.276358+0000","last_peered":"2024-03-30T13:18:15.276358+0000","last_clean":"2024-03-30T13:18:15.276358+0000","last_became_active":"2024-03-30T13:17:44.937731+0000","last_became_peered":"2024-03-30T13:17:44.937731+0000","last_unstale":"2024-03-30T13:18:15.276358+0000","last_undegraded":"2024-03-30T13:18:15.276358+0000","last_fullsized":"2024-03-30T13:18:15.276358+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:17:32.544446+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:17:32.544446+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:33.043891+0000","scrub_duration":331,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":95,"reported_epoch":626,"state":"active+clean","last_fresh":"2024-03-30T13:18:15.362646+0000","last_change":"2024-03-30T13:17:44.938352+0000","last_active":"2024-03-30T13:18:15.362646+0000","last_peered":"2024-03-30T13:18:15.362646+0000","last_clean":"2024-03-30T13:18:15.362646+0000","last_became_active":"2024-03-30T13:17:44.937300+0000","last_became_peered":"2024-03-30T13:17:44.937300+0000","last_unstale":"2024-03-30T13:18:15.362646+0000","last_undegraded":"2024-03-30T13:18:15.362646+0000","last_fullsized":"2024-03-30T13:18:15.362646+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:17:13.218985+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:17:13.218985+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:22.528329+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":103,"reported_epoch":626,"state":"active+clean","last_fresh":"2024-03-30T13:18:15.362465+0000","last_change":"2024-03-30T13:18:01.576814+0000","last_active":"2024-03-30T13:18:15.362465+0000","last_peered":"2024-03-30T13:18:15.362465+0000","last_clean":"2024-03-30T13:18:15.362465+0000","last_became_active":"2024-03-30T13:18:01.087050+0000","last_became_peered":"2024-03-30T13:18:01.087050+0000","last_unstale":"2024-03-30T13:18:15.362465+0000","last_undegraded":"2024-03-30T13:18:15.362465+0000","last_fullsized":"2024-03-30T13:18:15.362465+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:17:46.411360+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:17:46.411360+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:50.557860+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":272,"reported_epoch":626,"state":"active+clean","last_fresh":"2024-03-30T13:18:15.363094+0000","last_change":"2024-03-30T13:18:04.841148+0000","last_active":"2024-03-30T13:18:15.363094+0000","last_peered":"2024-03-30T13:18:15.363094+0000","last_clean":"2024-03-30T13:18:15.363094+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:18:15.363094+0000","last_undegraded":"2024-03-30T13:18:15.363094+0000","last_fullsized":"2024-03-30T13:18:15.363094+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:18:04.840923+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:18:04.840923+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:14.630038+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1434,"reported_epoch":626,"state":"active+clean","last_fresh":"2024-03-30T13:18:15.277994+0000","last_change":"2024-03-30T13:17:28.727465+0000","last_active":"2024-03-30T13:18:15.277994+0000","last_peered":"2024-03-30T13:18:15.277994+0000","last_clean":"2024-03-30T13:18:15.277994+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:18:15.277994+0000","last_undegraded":"2024-03-30T13:18:15.277994+0000","last_fullsized":"2024-03-30T13:18:15.277994+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:17:28.727219+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:12:15.741379+0000","last_clean_scrub_stamp":"2024-03-30T13:17:28.727219+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:37.241722+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":121,"reported_epoch":626,"state":"active+clean","last_fresh":"2024-03-30T13:18:15.362500+0000","last_change":"2024-03-30T13:17:44.936542+0000","last_active":"2024-03-30T13:18:15.362500+0000","last_peered":"2024-03-30T13:18:15.362500+0000","last_clean":"2024-03-30T13:18:15.362500+0000","last_became_active":"2024-03-30T13:17:44.935396+0000","last_became_peered":"2024-03-30T13:17:44.935396+0000","last_unstale":"2024-03-30T13:18:15.362500+0000","last_undegraded":"2024-03-30T13:18:15.362500+0000","last_fullsized":"2024-03-30T13:18:15.362500+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:16:57.724219+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:16:57.724219+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:23.701163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":675,"reported_epoch":626,"state":"active+clean","last_fresh":"2024-03-30T13:18:15.360255+0000","last_change":"2024-03-30T13:18:01.446426+0000","last_active":"2024-03-30T13:18:15.360255+0000","last_peered":"2024-03-30T13:18:15.360255+0000","last_clean":"2024-03-30T13:18:15.360255+0000","last_became_active":"2024-03-30T13:18:01.084237+0000","last_became_peered":"2024-03-30T13:18:01.084237+0000","last_unstale":"2024-03-30T13:18:15.360255+0000","last_undegraded":"2024-03-30T13:18:15.360255+0000","last_fullsized":"2024-03-30T13:18:15.360255+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:17:35.661077+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:17:35.661077+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:42.340736+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1579,"reported_epoch":626,"state":"active+clean","last_fresh":"2024-03-30T13:18:15.277783+0000","last_change":"2024-03-30T13:18:14.814370+0000","last_active":"2024-03-30T13:18:15.277783+0000","last_peered":"2024-03-30T13:18:15.277783+0000","last_clean":"2024-03-30T13:18:15.277783+0000","last_became_active":"2024-03-30T13:18:14.277074+0000","last_became_peered":"2024-03-30T13:18:14.277074+0000","last_unstale":"2024-03-30T13:18:15.277783+0000","last_undegraded":"2024-03-30T13:18:15.277783+0000","last_fullsized":"2024-03-30T13:18:15.277783+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:18:14.814123+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:18:14.814123+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:32.802268+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":81210368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1186,"reported_epoch":626,"state":"active+clean","last_fresh":"2024-03-30T13:18:15.276259+0000","last_change":"2024-03-30T13:18:01.111267+0000","last_active":"2024-03-30T13:18:15.276259+0000","last_peered":"2024-03-30T13:18:15.276259+0000","last_clean":"2024-03-30T13:18:15.276259+0000","last_became_active":"2024-03-30T13:18:01.079800+0000","last_became_peered":"2024-03-30T13:18:01.079800+0000","last_unstale":"2024-03-30T13:18:15.276259+0000","last_undegraded":"2024-03-30T13:18:15.276259+0000","last_fullsized":"2024-03-30T13:18:15.276259+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:17:36.151638+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_clean_scrub_stamp":"2024-03-30T13:17:36.151638+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:51.109693+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":29,"reported_epoch":626,"state":"active+clean","last_fresh":"2024-03-30T13:18:15.362774+0000","last_change":"2024-03-30T13:18:01.181380+0000","last_active":"2024-03-30T13:18:15.362774+0000","last_peered":"2024-03-30T13:18:15.362774+0000","last_clean":"2024-03-30T13:18:15.362774+0000","last_became_active":"2024-03-30T13:18:01.086819+0000","last_became_peered":"2024-03-30T13:18:01.086819+0000","last_unstale":"2024-03-30T13:18:15.362774+0000","last_undegraded":"2024-03-30T13:18:15.362774+0000","last_fullsized":"2024-03-30T13:18:15.362774+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_clean_scrub_stamp":"2024-03-30T13:17:50.531859+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:15.405349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":35,"reported_epoch":626,"state":"active+clean","last_fresh":"2024-03-30T13:18:15.362605+0000","last_change":"2024-03-30T13:18:01.326949+0000","last_active":"2024-03-30T13:18:15.362605+0000","last_peered":"2024-03-30T13:18:15.362605+0000","last_clean":"2024-03-30T13:18:15.362605+0000","last_became_active":"2024-03-30T13:18:01.097329+0000","last_became_peered":"2024-03-30T13:18:01.097329+0000","last_unstale":"2024-03-30T13:18:15.362605+0000","last_undegraded":"2024-03-30T13:18:15.362605+0000","last_fullsized":"2024-03-30T13:18:15.362605+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:17:40.468714+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:17:40.468714+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:58.731595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":285,"reported_epoch":626,"state":"active+clean","last_fresh":"2024-03-30T13:18:15.363069+0000","last_change":"2024-03-30T13:17:40.956701+0000","last_active":"2024-03-30T13:18:15.363069+0000","last_peered":"2024-03-30T13:18:15.363069+0000","last_clean":"2024-03-30T13:18:15.363069+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:18:15.363069+0000","last_undegraded":"2024-03-30T13:18:15.363069+0000","last_fullsized":"2024-03-30T13:18:15.363069+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:17:40.956450+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:17:40.956450+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:59.963567+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":87,"reported_epoch":626,"state":"active+clean","last_fresh":"2024-03-30T13:18:15.362905+0000","last_change":"2024-03-30T13:17:44.936616+0000","last_active":"2024-03-30T13:18:15.362905+0000","last_peered":"2024-03-30T13:18:15.362905+0000","last_clean":"2024-03-30T13:18:15.362905+0000","last_became_active":"2024-03-30T13:17:44.935038+0000","last_became_peered":"2024-03-30T13:17:44.935038+0000","last_unstale":"2024-03-30T13:18:15.362905+0000","last_undegraded":"2024-03-30T13:18:15.362905+0000","last_fullsized":"2024-03-30T13:18:15.362905+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:17:19.945177+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:17:19.945177+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:33.414637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":285,"reported_epoch":626,"state":"active+clean","last_fresh":"2024-03-30T13:18:15.363081+0000","last_change":"2024-03-30T13:18:10.804325+0000","last_active":"2024-03-30T13:18:15.363081+0000","last_peered":"2024-03-30T13:18:15.363081+0000","last_clean":"2024-03-30T13:18:15.363081+0000","last_became_active":"2024-03-30T13:18:01.090343+0000","last_became_peered":"2024-03-30T13:18:01.090343+0000","last_unstale":"2024-03-30T13:18:15.363081+0000","last_undegraded":"2024-03-30T13:18:15.363081+0000","last_fullsized":"2024-03-30T13:18:15.363081+0000","mapping_epoch":622,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":623,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:18:10.804054+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:18:10.804054+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:35.342726+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":642,"reported_epoch":626,"state":"active+clean","last_fresh":"2024-03-30T13:18:15.360316+0000","last_change":"2024-03-30T13:17:58.851110+0000","last_active":"2024-03-30T13:18:15.360316+0000","last_peered":"2024-03-30T13:18:15.360316+0000","last_clean":"2024-03-30T13:18:15.360316+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:18:15.360316+0000","last_undegraded":"2024-03-30T13:18:15.360316+0000","last_fullsized":"2024-03-30T13:18:15.360316+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:17:58.850996+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:15.607294+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":262168862,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":113467392,"data_stored":113467392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157487,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64968,"kb_used_data":27316,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94306872,"statfs":{"total":96636764160,"available":96570236928,"internally_reserved":0,"allocated":27971584,"data_stored":23775050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":559,"seq":2400886718490,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56664,"kb_used_data":19020,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94315176,"statfs":{"total":96636764160,"available":96578740224,"internally_reserved":0,"allocated":19476480,"data_stored":15279946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12639,"internal_metadata":38522529},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509717,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50452,"kb_used_data":15104,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94321388,"statfs":{"total":96636764160,"available":96585101312,"internally_reserved":0,"allocated":15466496,"data_stored":11269962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375998996,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50268,"kb_used_data":13064,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94321572,"statfs":{"total":96636764160,"available":96585289728,"internally_reserved":0,"allocated":13377536,"data_stored":9181002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509718,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61412,"kb_used_data":26064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94310428,"statfs":{"total":96636764160,"available":96573878272,"internally_reserved":0,"allocated":26689536,"data_stored":22493002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":604,"seq":2594160246794,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64040,"kb_used_data":26388,"kb_used_omap":13,"kb_used_meta":37618,"kb_avail":94307800,"statfs":{"total":96636764160,"available":96571187200,"internally_reserved":0,"allocated":27021312,"data_stored":22824778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14282,"internal_metadata":38520886},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147427,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52688,"kb_used_data":20992,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94319152,"statfs":{"total":96636764160,"available":96582811648,"internally_reserved":0,"allocated":21495808,"data_stored":17299274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19046400,"data_stored":19046400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7823360,"data_stored":7823360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":13852672,"data_stored":13852672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":19378176,"data_stored":19378176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:18:25.739 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:18:25.739 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:18:25.739 INFO:tasks.thrashosds.thrasher:Reviving osd 6 2024-03-30T13:18:25.739 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-30T13:18:25.739 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-30T13:18:25.742 INFO:tasks.ceph.osd.6:Started 2024-03-30T13:18:25.742 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-30T13:18:25.807 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T13:18:25.809+0000 7fe19d51f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:25.815 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T13:18:25.817+0000 7fe19d51f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:25.815 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T13:18:25.817+0000 7fe19d51f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:25.863 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T13:18:25.863 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T13:18:25.863 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T13:18:25.863 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T13:18:25.863 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T13:18:25.863 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T13:18:25.863 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-03-30T13:18:25.863 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-03-30T13:18:25.863 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-03-30T13:18:25.863 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-03-30T13:18:25.864 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-03-30T13:18:25.864 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T13:18:25.865 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:18:25.866 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-03-30T13:18:26.348 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T13:18:26.349+0000 7fe19d51f7c0 -1 Falling back to public interface 2024-03-30T13:18:30.172 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T13:18:30.173+0000 7fe19d51f7c0 -1 osd.6 621 log_to_monitors true 2024-03-30T13:18:30.415 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T13:18:30.417+0000 7fe1952f4640 -1 osd.6 621 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:18:30.866 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-30T13:18:31.000 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T13:18:31.000 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T13:18:31.000 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T13:18:31.000 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T13:18:31.013 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:18:31.014 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-03-30T13:18:31.014 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 6 2024-03-30T13:18:31.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:31.136+0000 7f6bcb55c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:31.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:31.152+0000 7f6bcb55c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:31.338 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:18:31.336+0000 7fbb38f08640 -1 mon.a@0(leader).osd e628 definitely_dead 0 2024-03-30T13:18:31.390 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.6. 2024-03-30T13:18:31.415 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-03-30T13:18:31.415 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:18:31.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:31.524+0000 7fedf94d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:31.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:31.540+0000 7fedf94d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:32.396 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:18:32.425 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:18:32.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:32.540+0000 7fc4c89cf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:32.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:32.556+0000 7fc4c89cf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:33.429 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:18:33.451 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:18:33.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:33.568+0000 7fdb96d39640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:33.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:33.584+0000 7fdb96d39640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:34.441 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:18:34.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:18:34.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:34.592+0000 7f30abe63640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:34.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:34.608+0000 7f30abe63640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:35.445 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:18:35.476 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:18:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:35.588+0000 7f01adc36640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:35.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:35.604+0000 7f01adc36640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:36.467 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:18:36.491 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:18:36.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:36.604+0000 7fe042125640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:36.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:36.620+0000 7fe042125640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:37.477 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:18:37.504 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:18:37.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:37.616+0000 7f097b9b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:37.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:37.632+0000 7f097b9b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:38.493 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:18:38.516 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:18:38.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:38.632+0000 7fae42ae5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:38.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:38.648+0000 7fae42ae5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:39.490 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:18:39.514 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:18:39.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:39.624+0000 7fa18701d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:39.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:39.644+0000 7fa18701d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:40.508 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:18:40.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:18:40.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:40.644+0000 7f874c2a0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:40.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:40.660+0000 7f874c2a0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:41.510 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:18:41.539 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:18:41.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:41.656+0000 7f1a09db0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:41.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:41.672+0000 7f1a09db0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:42.521 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:18:42.550 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:18:42.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:42.668+0000 7fdf32609640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:42.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:42.684+0000 7fdf32609640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:43.545 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:18:43.568 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:18:43.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:43.680+0000 7f822aac0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:43.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:43.696+0000 7f822aac0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:44.548 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:18:44.574 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:18:44.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:44.692+0000 7f924d0e3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:44.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:44.708+0000 7f924d0e3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:45.558 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:18:45.583 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:18:45.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:45.704+0000 7f25300b3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:45.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:45.720+0000 7f25300b3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:46.562 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:18:46.587 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:18:46.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:46.704+0000 7fcdb165f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:46.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:46.720+0000 7fcdb165f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:47.583 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:18:47.611 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:18:47.611 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:18:47.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:47.728+0000 7f357e7e9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:47.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:47.740+0000 7f357e7e9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:47.920 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:18:47.921 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:18:47.938 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1335,"stamp":"2024-03-30T13:18:47.588473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":230185570,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":445008,"kb_used_data":155720,"kb_used_omap":61,"kb_used_meta":289090,"kb_avail":754529712,"statfs":{"total":773094113280,"available":772638425088,"internally_reserved":0,"allocated":159457280,"data_stored":125036285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":62546,"internal_metadata":296029102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.042665"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1486,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-03-30T13:18:45.786479+0000","last_change":"2024-03-30T13:18:32.417584+0000","last_active":"2024-03-30T13:18:45.786479+0000","last_peered":"2024-03-30T13:18:45.786479+0000","last_clean":"2024-03-30T13:18:45.786479+0000","last_became_active":"2024-03-30T13:18:32.416374+0000","last_became_peered":"2024-03-30T13:18:32.416374+0000","last_unstale":"2024-03-30T13:18:45.786479+0000","last_undegraded":"2024-03-30T13:18:45.786479+0000","last_fullsized":"2024-03-30T13:18:45.786479+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:17:53.569165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:56.559697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1367,"reported_epoch":640,"state":"active+clean","last_fresh":"2024-03-30T13:18:42.926168+0000","last_change":"2024-03-30T13:18:33.123163+0000","last_active":"2024-03-30T13:18:42.926168+0000","last_peered":"2024-03-30T13:18:42.926168+0000","last_clean":"2024-03-30T13:18:42.926168+0000","last_became_active":"2024-03-30T13:17:44.937731+0000","last_became_peered":"2024-03-30T13:17:44.937731+0000","last_unstale":"2024-03-30T13:18:42.926168+0000","last_undegraded":"2024-03-30T13:18:42.926168+0000","last_fullsized":"2024-03-30T13:18:42.926168+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:18:33.122892+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:18:33.122892+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:37.289093+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":132,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-03-30T13:18:43.547905+0000","last_change":"2024-03-30T13:18:22.850213+0000","last_active":"2024-03-30T13:18:43.547905+0000","last_peered":"2024-03-30T13:18:43.547905+0000","last_clean":"2024-03-30T13:18:43.547905+0000","last_became_active":"2024-03-30T13:17:44.937300+0000","last_became_peered":"2024-03-30T13:17:44.937300+0000","last_unstale":"2024-03-30T13:18:43.547905+0000","last_undegraded":"2024-03-30T13:18:43.547905+0000","last_fullsized":"2024-03-30T13:18:43.547905+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:18:22.849961+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:18:22.849961+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:26.050541+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":138,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-03-30T13:18:43.547897+0000","last_change":"2024-03-30T13:18:32.422320+0000","last_active":"2024-03-30T13:18:43.547897+0000","last_peered":"2024-03-30T13:18:43.547897+0000","last_clean":"2024-03-30T13:18:43.547897+0000","last_became_active":"2024-03-30T13:18:32.421168+0000","last_became_peered":"2024-03-30T13:18:32.421168+0000","last_unstale":"2024-03-30T13:18:43.547897+0000","last_undegraded":"2024-03-30T13:18:43.547897+0000","last_fullsized":"2024-03-30T13:18:43.547897+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:17:46.411360+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:17:46.411360+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:54.421917+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":309,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T13:18:46.624266+0000","last_change":"2024-03-30T13:18:04.841148+0000","last_active":"2024-03-30T13:18:46.624266+0000","last_peered":"2024-03-30T13:18:46.624266+0000","last_clean":"2024-03-30T13:18:46.624266+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:18:46.624266+0000","last_undegraded":"2024-03-30T13:18:46.624266+0000","last_fullsized":"2024-03-30T13:18:46.624266+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:18:04.840923+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:18:04.840923+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:14.630038+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1477,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T13:18:46.571219+0000","last_change":"2024-03-30T13:18:42.729125+0000","last_active":"2024-03-30T13:18:46.571219+0000","last_peered":"2024-03-30T13:18:46.571219+0000","last_clean":"2024-03-30T13:18:46.571219+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:18:46.571219+0000","last_undegraded":"2024-03-30T13:18:46.571219+0000","last_fullsized":"2024-03-30T13:18:46.571219+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_clean_scrub_stamp":"2024-03-30T13:18:42.729016+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:03.586884+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":158,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-03-30T13:18:43.547897+0000","last_change":"2024-03-30T13:18:27.807956+0000","last_active":"2024-03-30T13:18:43.547897+0000","last_peered":"2024-03-30T13:18:43.547897+0000","last_clean":"2024-03-30T13:18:43.547897+0000","last_became_active":"2024-03-30T13:17:44.935396+0000","last_became_peered":"2024-03-30T13:17:44.935396+0000","last_unstale":"2024-03-30T13:18:43.547897+0000","last_undegraded":"2024-03-30T13:18:43.547897+0000","last_fullsized":"2024-03-30T13:18:43.547897+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:18:27.807804+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:18:27.807804+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:56.928075+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":713,"reported_epoch":642,"state":"active+clean","last_fresh":"2024-03-30T13:18:44.554159+0000","last_change":"2024-03-30T13:18:32.414854+0000","last_active":"2024-03-30T13:18:44.554159+0000","last_peered":"2024-03-30T13:18:44.554159+0000","last_clean":"2024-03-30T13:18:44.554159+0000","last_became_active":"2024-03-30T13:18:32.413904+0000","last_became_peered":"2024-03-30T13:18:32.413904+0000","last_unstale":"2024-03-30T13:18:44.554159+0000","last_undegraded":"2024-03-30T13:18:44.554159+0000","last_fullsized":"2024-03-30T13:18:44.554159+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:17:35.661077+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:17:35.661077+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:59.785888+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1615,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T13:18:46.571059+0000","last_change":"2024-03-30T13:18:14.814370+0000","last_active":"2024-03-30T13:18:46.571059+0000","last_peered":"2024-03-30T13:18:46.571059+0000","last_clean":"2024-03-30T13:18:46.571059+0000","last_became_active":"2024-03-30T13:18:14.277074+0000","last_became_peered":"2024-03-30T13:18:14.277074+0000","last_unstale":"2024-03-30T13:18:46.571059+0000","last_undegraded":"2024-03-30T13:18:46.571059+0000","last_fullsized":"2024-03-30T13:18:46.571059+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:18:14.814123+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:18:14.814123+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:32.802268+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":81210368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1221,"reported_epoch":640,"state":"active+clean","last_fresh":"2024-03-30T13:18:42.925650+0000","last_change":"2024-03-30T13:18:32.417767+0000","last_active":"2024-03-30T13:18:42.925650+0000","last_peered":"2024-03-30T13:18:42.925650+0000","last_clean":"2024-03-30T13:18:42.925650+0000","last_became_active":"2024-03-30T13:18:32.416688+0000","last_became_peered":"2024-03-30T13:18:32.416688+0000","last_unstale":"2024-03-30T13:18:42.925650+0000","last_undegraded":"2024-03-30T13:18:42.925650+0000","last_fullsized":"2024-03-30T13:18:42.925650+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:17:36.151638+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_clean_scrub_stamp":"2024-03-30T13:17:36.151638+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:03.568081+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1463,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-03-30T13:18:45.786301+0000","last_change":"2024-03-30T13:18:32.414752+0000","last_active":"2024-03-30T13:18:45.786301+0000","last_peered":"2024-03-30T13:18:45.786301+0000","last_clean":"2024-03-30T13:18:45.786301+0000","last_became_active":"2024-03-30T13:18:32.412897+0000","last_became_peered":"2024-03-30T13:18:32.412897+0000","last_unstale":"2024-03-30T13:18:45.786301+0000","last_undegraded":"2024-03-30T13:18:45.786301+0000","last_fullsized":"2024-03-30T13:18:45.786301+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_clean_scrub_stamp":"2024-03-30T13:17:50.531859+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:15.674072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1514,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-03-30T13:18:45.786267+0000","last_change":"2024-03-30T13:18:32.421100+0000","last_active":"2024-03-30T13:18:45.786267+0000","last_peered":"2024-03-30T13:18:45.786267+0000","last_clean":"2024-03-30T13:18:45.786267+0000","last_became_active":"2024-03-30T13:18:32.419902+0000","last_became_peered":"2024-03-30T13:18:32.419902+0000","last_unstale":"2024-03-30T13:18:45.786267+0000","last_undegraded":"2024-03-30T13:18:45.786267+0000","last_fullsized":"2024-03-30T13:18:45.786267+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:17:40.468714+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:17:40.468714+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:56.619526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":321,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T13:18:46.624490+0000","last_change":"2024-03-30T13:17:40.956701+0000","last_active":"2024-03-30T13:18:46.624490+0000","last_peered":"2024-03-30T13:18:46.624490+0000","last_clean":"2024-03-30T13:18:46.624490+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:18:46.624490+0000","last_undegraded":"2024-03-30T13:18:46.624490+0000","last_fullsized":"2024-03-30T13:18:46.624490+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:17:40.956450+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:17:40.956450+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:59.963567+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":124,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-03-30T13:18:43.547897+0000","last_change":"2024-03-30T13:18:35.867309+0000","last_active":"2024-03-30T13:18:43.547897+0000","last_peered":"2024-03-30T13:18:43.547897+0000","last_clean":"2024-03-30T13:18:43.547897+0000","last_became_active":"2024-03-30T13:17:44.935038+0000","last_became_peered":"2024-03-30T13:17:44.935038+0000","last_unstale":"2024-03-30T13:18:43.547897+0000","last_undegraded":"2024-03-30T13:18:43.547897+0000","last_fullsized":"2024-03-30T13:18:43.547897+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:18:35.867018+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:18:35.867018+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:35.869590+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":327,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T13:18:46.624053+0000","last_change":"2024-03-30T13:18:32.416876+0000","last_active":"2024-03-30T13:18:46.624053+0000","last_peered":"2024-03-30T13:18:46.624053+0000","last_clean":"2024-03-30T13:18:46.624053+0000","last_became_active":"2024-03-30T13:18:32.415413+0000","last_became_peered":"2024-03-30T13:18:32.415413+0000","last_unstale":"2024-03-30T13:18:46.624053+0000","last_undegraded":"2024-03-30T13:18:46.624053+0000","last_fullsized":"2024-03-30T13:18:46.624053+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:18:10.804054+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:18:10.804054+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:16.441042+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":674,"reported_epoch":642,"state":"active+clean","last_fresh":"2024-03-30T13:18:44.554312+0000","last_change":"2024-03-30T13:17:58.851110+0000","last_active":"2024-03-30T13:18:44.554312+0000","last_peered":"2024-03-30T13:18:44.554312+0000","last_clean":"2024-03-30T13:18:44.554312+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:18:44.554312+0000","last_undegraded":"2024-03-30T13:18:44.554312+0000","last_fullsized":"2024-03-30T13:18:44.554312+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:17:58.850996+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:15.607294+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":230185570,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":629,"seq":2701534429189,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59528,"kb_used_data":22776,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94312312,"statfs":{"total":96636764160,"available":96575807488,"internally_reserved":0,"allocated":23322624,"data_stored":19012361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157492,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65184,"kb_used_data":27532,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94306656,"statfs":{"total":96636764160,"available":96570015744,"internally_reserved":0,"allocated":28192768,"data_stored":23875704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":559,"seq":2400886718495,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56832,"kb_used_data":19188,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94315008,"statfs":{"total":96636764160,"available":96578568192,"internally_reserved":0,"allocated":19648512,"data_stored":15358602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12639,"internal_metadata":38522529},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509722,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49308,"kb_used_data":13968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94322532,"statfs":{"total":96636764160,"available":96586272768,"internally_reserved":0,"allocated":14303232,"data_stored":10006571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375999001,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50456,"kb_used_data":13256,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94321384,"statfs":{"total":96636764160,"available":96585097216,"internally_reserved":0,"allocated":13574144,"data_stored":9270682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509723,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56480,"kb_used_data":21132,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315360,"statfs":{"total":96636764160,"available":96578928640,"internally_reserved":0,"allocated":21639168,"data_stored":17349258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":604,"seq":2594160246799,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55336,"kb_used_data":17688,"kb_used_omap":13,"kb_used_meta":37618,"kb_avail":94316504,"statfs":{"total":96636764160,"available":96580100096,"internally_reserved":0,"allocated":18112512,"data_stored":13815851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14282,"internal_metadata":38520886},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147432,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51884,"kb_used_data":20180,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94319956,"statfs":{"total":96636764160,"available":96583634944,"internally_reserved":0,"allocated":20664320,"data_stored":16347256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:18:47.940 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:18:48.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:48.056+0000 7f9e85441640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:48.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:48.268+0000 7f9e7f7fe640 0 --2- 172.21.15.67:0/749563852 >> v2:172.21.15.67:3300/0 conn(0x7f9e80140be0 0x7f9e8013d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-03-30T13:18:48.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:48.272+0000 7f9e85441640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:48.452 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:18:48.453 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:18:48.471 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1336,"stamp":"2024-03-30T13:18:48.185804+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":230185570,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":445008,"kb_used_data":155720,"kb_used_omap":61,"kb_used_meta":289090,"kb_avail":754529712,"statfs":{"total":773094113280,"available":772638425088,"internally_reserved":0,"allocated":159457280,"data_stored":125036285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":62546,"internal_metadata":296029102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001037"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1486,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-03-30T13:18:45.786479+0000","last_change":"2024-03-30T13:18:32.417584+0000","last_active":"2024-03-30T13:18:45.786479+0000","last_peered":"2024-03-30T13:18:45.786479+0000","last_clean":"2024-03-30T13:18:45.786479+0000","last_became_active":"2024-03-30T13:18:32.416374+0000","last_became_peered":"2024-03-30T13:18:32.416374+0000","last_unstale":"2024-03-30T13:18:45.786479+0000","last_undegraded":"2024-03-30T13:18:45.786479+0000","last_fullsized":"2024-03-30T13:18:45.786479+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:17:53.569165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:56.559697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1367,"reported_epoch":640,"state":"active+clean","last_fresh":"2024-03-30T13:18:42.926168+0000","last_change":"2024-03-30T13:18:33.123163+0000","last_active":"2024-03-30T13:18:42.926168+0000","last_peered":"2024-03-30T13:18:42.926168+0000","last_clean":"2024-03-30T13:18:42.926168+0000","last_became_active":"2024-03-30T13:17:44.937731+0000","last_became_peered":"2024-03-30T13:17:44.937731+0000","last_unstale":"2024-03-30T13:18:42.926168+0000","last_undegraded":"2024-03-30T13:18:42.926168+0000","last_fullsized":"2024-03-30T13:18:42.926168+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:18:33.122892+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:18:33.122892+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:37.289093+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":132,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-03-30T13:18:43.547905+0000","last_change":"2024-03-30T13:18:22.850213+0000","last_active":"2024-03-30T13:18:43.547905+0000","last_peered":"2024-03-30T13:18:43.547905+0000","last_clean":"2024-03-30T13:18:43.547905+0000","last_became_active":"2024-03-30T13:17:44.937300+0000","last_became_peered":"2024-03-30T13:17:44.937300+0000","last_unstale":"2024-03-30T13:18:43.547905+0000","last_undegraded":"2024-03-30T13:18:43.547905+0000","last_fullsized":"2024-03-30T13:18:43.547905+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:18:22.849961+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:18:22.849961+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:26.050541+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":138,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-03-30T13:18:43.547897+0000","last_change":"2024-03-30T13:18:32.422320+0000","last_active":"2024-03-30T13:18:43.547897+0000","last_peered":"2024-03-30T13:18:43.547897+0000","last_clean":"2024-03-30T13:18:43.547897+0000","last_became_active":"2024-03-30T13:18:32.421168+0000","last_became_peered":"2024-03-30T13:18:32.421168+0000","last_unstale":"2024-03-30T13:18:43.547897+0000","last_undegraded":"2024-03-30T13:18:43.547897+0000","last_fullsized":"2024-03-30T13:18:43.547897+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:17:46.411360+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:17:46.411360+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:54.421917+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":309,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T13:18:46.624266+0000","last_change":"2024-03-30T13:18:04.841148+0000","last_active":"2024-03-30T13:18:46.624266+0000","last_peered":"2024-03-30T13:18:46.624266+0000","last_clean":"2024-03-30T13:18:46.624266+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:18:46.624266+0000","last_undegraded":"2024-03-30T13:18:46.624266+0000","last_fullsized":"2024-03-30T13:18:46.624266+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:18:04.840923+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:18:04.840923+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:14.630038+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1477,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T13:18:46.571219+0000","last_change":"2024-03-30T13:18:42.729125+0000","last_active":"2024-03-30T13:18:46.571219+0000","last_peered":"2024-03-30T13:18:46.571219+0000","last_clean":"2024-03-30T13:18:46.571219+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:18:46.571219+0000","last_undegraded":"2024-03-30T13:18:46.571219+0000","last_fullsized":"2024-03-30T13:18:46.571219+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_clean_scrub_stamp":"2024-03-30T13:18:42.729016+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:03.586884+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":158,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-03-30T13:18:43.547897+0000","last_change":"2024-03-30T13:18:27.807956+0000","last_active":"2024-03-30T13:18:43.547897+0000","last_peered":"2024-03-30T13:18:43.547897+0000","last_clean":"2024-03-30T13:18:43.547897+0000","last_became_active":"2024-03-30T13:17:44.935396+0000","last_became_peered":"2024-03-30T13:17:44.935396+0000","last_unstale":"2024-03-30T13:18:43.547897+0000","last_undegraded":"2024-03-30T13:18:43.547897+0000","last_fullsized":"2024-03-30T13:18:43.547897+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:18:27.807804+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:18:27.807804+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:56.928075+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":713,"reported_epoch":642,"state":"active+clean","last_fresh":"2024-03-30T13:18:44.554159+0000","last_change":"2024-03-30T13:18:32.414854+0000","last_active":"2024-03-30T13:18:44.554159+0000","last_peered":"2024-03-30T13:18:44.554159+0000","last_clean":"2024-03-30T13:18:44.554159+0000","last_became_active":"2024-03-30T13:18:32.413904+0000","last_became_peered":"2024-03-30T13:18:32.413904+0000","last_unstale":"2024-03-30T13:18:44.554159+0000","last_undegraded":"2024-03-30T13:18:44.554159+0000","last_fullsized":"2024-03-30T13:18:44.554159+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:17:35.661077+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:17:35.661077+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:59.785888+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1615,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T13:18:46.571059+0000","last_change":"2024-03-30T13:18:14.814370+0000","last_active":"2024-03-30T13:18:46.571059+0000","last_peered":"2024-03-30T13:18:46.571059+0000","last_clean":"2024-03-30T13:18:46.571059+0000","last_became_active":"2024-03-30T13:18:14.277074+0000","last_became_peered":"2024-03-30T13:18:14.277074+0000","last_unstale":"2024-03-30T13:18:46.571059+0000","last_undegraded":"2024-03-30T13:18:46.571059+0000","last_fullsized":"2024-03-30T13:18:46.571059+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:18:14.814123+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:18:14.814123+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:32.802268+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":81210368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1221,"reported_epoch":640,"state":"active+clean","last_fresh":"2024-03-30T13:18:42.925650+0000","last_change":"2024-03-30T13:18:32.417767+0000","last_active":"2024-03-30T13:18:42.925650+0000","last_peered":"2024-03-30T13:18:42.925650+0000","last_clean":"2024-03-30T13:18:42.925650+0000","last_became_active":"2024-03-30T13:18:32.416688+0000","last_became_peered":"2024-03-30T13:18:32.416688+0000","last_unstale":"2024-03-30T13:18:42.925650+0000","last_undegraded":"2024-03-30T13:18:42.925650+0000","last_fullsized":"2024-03-30T13:18:42.925650+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:17:36.151638+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_clean_scrub_stamp":"2024-03-30T13:17:36.151638+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:03.568081+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1463,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-03-30T13:18:45.786301+0000","last_change":"2024-03-30T13:18:32.414752+0000","last_active":"2024-03-30T13:18:45.786301+0000","last_peered":"2024-03-30T13:18:45.786301+0000","last_clean":"2024-03-30T13:18:45.786301+0000","last_became_active":"2024-03-30T13:18:32.412897+0000","last_became_peered":"2024-03-30T13:18:32.412897+0000","last_unstale":"2024-03-30T13:18:45.786301+0000","last_undegraded":"2024-03-30T13:18:45.786301+0000","last_fullsized":"2024-03-30T13:18:45.786301+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_clean_scrub_stamp":"2024-03-30T13:17:50.531859+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:15.674072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1514,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-03-30T13:18:45.786267+0000","last_change":"2024-03-30T13:18:32.421100+0000","last_active":"2024-03-30T13:18:45.786267+0000","last_peered":"2024-03-30T13:18:45.786267+0000","last_clean":"2024-03-30T13:18:45.786267+0000","last_became_active":"2024-03-30T13:18:32.419902+0000","last_became_peered":"2024-03-30T13:18:32.419902+0000","last_unstale":"2024-03-30T13:18:45.786267+0000","last_undegraded":"2024-03-30T13:18:45.786267+0000","last_fullsized":"2024-03-30T13:18:45.786267+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:17:40.468714+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:17:40.468714+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:56.619526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":321,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T13:18:46.624490+0000","last_change":"2024-03-30T13:17:40.956701+0000","last_active":"2024-03-30T13:18:46.624490+0000","last_peered":"2024-03-30T13:18:46.624490+0000","last_clean":"2024-03-30T13:18:46.624490+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:18:46.624490+0000","last_undegraded":"2024-03-30T13:18:46.624490+0000","last_fullsized":"2024-03-30T13:18:46.624490+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:17:40.956450+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:17:40.956450+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:59.963567+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":124,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-03-30T13:18:43.547897+0000","last_change":"2024-03-30T13:18:35.867309+0000","last_active":"2024-03-30T13:18:43.547897+0000","last_peered":"2024-03-30T13:18:43.547897+0000","last_clean":"2024-03-30T13:18:43.547897+0000","last_became_active":"2024-03-30T13:17:44.935038+0000","last_became_peered":"2024-03-30T13:17:44.935038+0000","last_unstale":"2024-03-30T13:18:43.547897+0000","last_undegraded":"2024-03-30T13:18:43.547897+0000","last_fullsized":"2024-03-30T13:18:43.547897+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:18:35.867018+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:18:35.867018+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:35.869590+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":327,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T13:18:46.624053+0000","last_change":"2024-03-30T13:18:32.416876+0000","last_active":"2024-03-30T13:18:46.624053+0000","last_peered":"2024-03-30T13:18:46.624053+0000","last_clean":"2024-03-30T13:18:46.624053+0000","last_became_active":"2024-03-30T13:18:32.415413+0000","last_became_peered":"2024-03-30T13:18:32.415413+0000","last_unstale":"2024-03-30T13:18:46.624053+0000","last_undegraded":"2024-03-30T13:18:46.624053+0000","last_fullsized":"2024-03-30T13:18:46.624053+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:18:10.804054+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:18:10.804054+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:16.441042+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":674,"reported_epoch":642,"state":"active+clean","last_fresh":"2024-03-30T13:18:44.554312+0000","last_change":"2024-03-30T13:17:58.851110+0000","last_active":"2024-03-30T13:18:44.554312+0000","last_peered":"2024-03-30T13:18:44.554312+0000","last_clean":"2024-03-30T13:18:44.554312+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:18:44.554312+0000","last_undegraded":"2024-03-30T13:18:44.554312+0000","last_fullsized":"2024-03-30T13:18:44.554312+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:17:58.850996+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:15.607294+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":230185570,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":629,"seq":2701534429189,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59528,"kb_used_data":22776,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94312312,"statfs":{"total":96636764160,"available":96575807488,"internally_reserved":0,"allocated":23322624,"data_stored":19012361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157492,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65184,"kb_used_data":27532,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94306656,"statfs":{"total":96636764160,"available":96570015744,"internally_reserved":0,"allocated":28192768,"data_stored":23875704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":559,"seq":2400886718495,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56832,"kb_used_data":19188,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94315008,"statfs":{"total":96636764160,"available":96578568192,"internally_reserved":0,"allocated":19648512,"data_stored":15358602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12639,"internal_metadata":38522529},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509722,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49308,"kb_used_data":13968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94322532,"statfs":{"total":96636764160,"available":96586272768,"internally_reserved":0,"allocated":14303232,"data_stored":10006571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375999001,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50456,"kb_used_data":13256,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94321384,"statfs":{"total":96636764160,"available":96585097216,"internally_reserved":0,"allocated":13574144,"data_stored":9270682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509723,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56480,"kb_used_data":21132,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315360,"statfs":{"total":96636764160,"available":96578928640,"internally_reserved":0,"allocated":21639168,"data_stored":17349258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":604,"seq":2594160246799,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55336,"kb_used_data":17688,"kb_used_omap":13,"kb_used_meta":37618,"kb_avail":94316504,"statfs":{"total":96636764160,"available":96580100096,"internally_reserved":0,"allocated":18112512,"data_stored":13815851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14282,"internal_metadata":38520886},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147432,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51884,"kb_used_data":20180,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94319956,"statfs":{"total":96636764160,"available":96583634944,"internally_reserved":0,"allocated":20664320,"data_stored":16347256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:18:48.473 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:18:48.473 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:18:48.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:48.588+0000 7f3b2c2c0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:48.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:48.604+0000 7f3b2c2c0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:48.780 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:18:48.780 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":645,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:18:47.581828+0000","last_up_change":"2024-03-30T13:18:31.382822+0000","last_in_change":"2024-03-30T13:18:31.382822+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":32,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":598,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":461,"last_clean_end":575,"up_from":581,"up_thru":629,"down_at":576,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6818","nonce":4030404750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6821","nonce":4030404750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6828","nonce":4030404750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6825","nonce":4030404750}]},"public_addr":"172.21.15.67:6818/4030404750","cluster_addr":"172.21.15.67:6821/4030404750","heartbeat_back_addr":"172.21.15.67:6828/4030404750","heartbeat_front_addr":"172.21.15.67:6825/4030404750","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":607,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":388,"up_from":393,"up_thru":624,"down_at":389,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6801","nonce":3837194718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6803","nonce":3837194718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6807","nonce":3837194718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6805","nonce":3837194718}]},"public_addr":"172.21.15.67:6801/3837194718","cluster_addr":"172.21.15.67:6803/3837194718","heartbeat_back_addr":"172.21.15.67:6807/3837194718","heartbeat_front_addr":"172.21.15.67:6805/3837194718","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":536,"last_clean_end":552,"up_from":559,"up_thru":629,"down_at":553,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3095958959},{"type":"v1","addr":"172.21.15.82:6803","nonce":3095958959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":3095958959},{"type":"v1","addr":"172.21.15.82:6807","nonce":3095958959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":3095958959},{"type":"v1","addr":"172.21.15.82:6815","nonce":3095958959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":3095958959},{"type":"v1","addr":"172.21.15.82:6811","nonce":3095958959}]},"public_addr":"172.21.15.82:6803/3095958959","cluster_addr":"172.21.15.82:6807/3095958959","heartbeat_back_addr":"172.21.15.82:6815/3095958959","heartbeat_front_addr":"172.21.15.82:6811/3095958959","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":488,"last_clean_end":506,"up_from":515,"up_thru":629,"down_at":507,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6817","nonce":292505453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6819","nonce":292505453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6823","nonce":292505453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6821","nonce":292505453}]},"public_addr":"172.21.15.82:6817/292505453","cluster_addr":"172.21.15.82:6819/292505453","heartbeat_back_addr":"172.21.15.82:6823/292505453","heartbeat_front_addr":"172.21.15.82:6821/292505453","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":372,"last_clean_end":621,"up_from":629,"up_thru":629,"down_at":622,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3888194455},{"type":"v1","addr":"172.21.15.82:6801","nonce":3888194455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3888194455},{"type":"v1","addr":"172.21.15.82:6806","nonce":3888194455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3888194455},{"type":"v1","addr":"172.21.15.82:6814","nonce":3888194455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3888194455},{"type":"v1","addr":"172.21.15.82:6810","nonce":3888194455}]},"public_addr":"172.21.15.82:6801/3888194455","cluster_addr":"172.21.15.82:6806/3888194455","heartbeat_back_addr":"172.21.15.82:6814/3888194455","heartbeat_front_addr":"172.21.15.82:6810/3888194455","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":419,"last_clean_end":597,"up_from":604,"up_thru":629,"down_at":598,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":3016824960},{"type":"v1","addr":"172.21.15.82:6825","nonce":3016824960}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":3016824960},{"type":"v1","addr":"172.21.15.82:6827","nonce":3016824960}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":3016824960},{"type":"v1","addr":"172.21.15.82:6831","nonce":3016824960}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":3016824960},{"type":"v1","addr":"172.21.15.82:6829","nonce":3016824960}]},"public_addr":"172.21.15.82:6825/3016824960","cluster_addr":"172.21.15.82:6827/3016824960","heartbeat_back_addr":"172.21.15.82:6831/3016824960","heartbeat_front_addr":"172.21.15.82:6829/3016824960","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:18:36.683094+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:18:42.615259+0000","dead_epoch":576},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:18:25.559437+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:18:37.671848+0000","dead_epoch":389},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:18:14.708642+0000","dead_epoch":553},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:18:41.029277+0000","dead_epoch":507},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:18:31.148522+0000","dead_epoch":622},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:18:12.490673+0000","dead_epoch":598}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:18:48.798 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:18:48.799 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:18:48.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:48.916+0000 7ff3aa3ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:48.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:48.932+0000 7ff3aa3ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:49.105 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:18:49.125 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:18:49.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:49.232+0000 7fe68956d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:49.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:49.248+0000 7fe68956d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:49.425 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:18:49.426 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:18:49.443 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:18:49.443 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:18:49.443 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:18:49.443 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:18:49.443 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 3, 5, 4, 1, 7, 6] 2024-03-30T13:18:49.443 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:18:49.444 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:18:49.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:49.556+0000 7fb3a0bca640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:49.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:49.568+0000 7fb3a0bca640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:18:49.739 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:18:49.755 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1336,"stamp":"2024-03-30T13:18:48.185804+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":230185570,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":445008,"kb_used_data":155720,"kb_used_omap":61,"kb_used_meta":289090,"kb_avail":754529712,"statfs":{"total":773094113280,"available":772638425088,"internally_reserved":0,"allocated":159457280,"data_stored":125036285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":62546,"internal_metadata":296029102},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001037"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1486,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-03-30T13:18:45.786479+0000","last_change":"2024-03-30T13:18:32.417584+0000","last_active":"2024-03-30T13:18:45.786479+0000","last_peered":"2024-03-30T13:18:45.786479+0000","last_clean":"2024-03-30T13:18:45.786479+0000","last_became_active":"2024-03-30T13:18:32.416374+0000","last_became_peered":"2024-03-30T13:18:32.416374+0000","last_unstale":"2024-03-30T13:18:45.786479+0000","last_undegraded":"2024-03-30T13:18:45.786479+0000","last_fullsized":"2024-03-30T13:18:45.786479+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:17:53.569165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:56.559697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1367,"reported_epoch":640,"state":"active+clean","last_fresh":"2024-03-30T13:18:42.926168+0000","last_change":"2024-03-30T13:18:33.123163+0000","last_active":"2024-03-30T13:18:42.926168+0000","last_peered":"2024-03-30T13:18:42.926168+0000","last_clean":"2024-03-30T13:18:42.926168+0000","last_became_active":"2024-03-30T13:17:44.937731+0000","last_became_peered":"2024-03-30T13:17:44.937731+0000","last_unstale":"2024-03-30T13:18:42.926168+0000","last_undegraded":"2024-03-30T13:18:42.926168+0000","last_fullsized":"2024-03-30T13:18:42.926168+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:18:33.122892+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:18:33.122892+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:37.289093+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":132,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-03-30T13:18:43.547905+0000","last_change":"2024-03-30T13:18:22.850213+0000","last_active":"2024-03-30T13:18:43.547905+0000","last_peered":"2024-03-30T13:18:43.547905+0000","last_clean":"2024-03-30T13:18:43.547905+0000","last_became_active":"2024-03-30T13:17:44.937300+0000","last_became_peered":"2024-03-30T13:17:44.937300+0000","last_unstale":"2024-03-30T13:18:43.547905+0000","last_undegraded":"2024-03-30T13:18:43.547905+0000","last_fullsized":"2024-03-30T13:18:43.547905+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:18:22.849961+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:18:22.849961+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:26.050541+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":138,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-03-30T13:18:43.547897+0000","last_change":"2024-03-30T13:18:32.422320+0000","last_active":"2024-03-30T13:18:43.547897+0000","last_peered":"2024-03-30T13:18:43.547897+0000","last_clean":"2024-03-30T13:18:43.547897+0000","last_became_active":"2024-03-30T13:18:32.421168+0000","last_became_peered":"2024-03-30T13:18:32.421168+0000","last_unstale":"2024-03-30T13:18:43.547897+0000","last_undegraded":"2024-03-30T13:18:43.547897+0000","last_fullsized":"2024-03-30T13:18:43.547897+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:17:46.411360+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:17:46.411360+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:54.421917+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":309,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T13:18:46.624266+0000","last_change":"2024-03-30T13:18:04.841148+0000","last_active":"2024-03-30T13:18:46.624266+0000","last_peered":"2024-03-30T13:18:46.624266+0000","last_clean":"2024-03-30T13:18:46.624266+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:18:46.624266+0000","last_undegraded":"2024-03-30T13:18:46.624266+0000","last_fullsized":"2024-03-30T13:18:46.624266+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:18:04.840923+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:18:04.840923+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:14.630038+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1477,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T13:18:46.571219+0000","last_change":"2024-03-30T13:18:42.729125+0000","last_active":"2024-03-30T13:18:46.571219+0000","last_peered":"2024-03-30T13:18:46.571219+0000","last_clean":"2024-03-30T13:18:46.571219+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:18:46.571219+0000","last_undegraded":"2024-03-30T13:18:46.571219+0000","last_fullsized":"2024-03-30T13:18:46.571219+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_clean_scrub_stamp":"2024-03-30T13:18:42.729016+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:03.586884+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":158,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-03-30T13:18:43.547897+0000","last_change":"2024-03-30T13:18:27.807956+0000","last_active":"2024-03-30T13:18:43.547897+0000","last_peered":"2024-03-30T13:18:43.547897+0000","last_clean":"2024-03-30T13:18:43.547897+0000","last_became_active":"2024-03-30T13:17:44.935396+0000","last_became_peered":"2024-03-30T13:17:44.935396+0000","last_unstale":"2024-03-30T13:18:43.547897+0000","last_undegraded":"2024-03-30T13:18:43.547897+0000","last_fullsized":"2024-03-30T13:18:43.547897+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:18:27.807804+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:18:27.807804+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:56.928075+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":713,"reported_epoch":642,"state":"active+clean","last_fresh":"2024-03-30T13:18:44.554159+0000","last_change":"2024-03-30T13:18:32.414854+0000","last_active":"2024-03-30T13:18:44.554159+0000","last_peered":"2024-03-30T13:18:44.554159+0000","last_clean":"2024-03-30T13:18:44.554159+0000","last_became_active":"2024-03-30T13:18:32.413904+0000","last_became_peered":"2024-03-30T13:18:32.413904+0000","last_unstale":"2024-03-30T13:18:44.554159+0000","last_undegraded":"2024-03-30T13:18:44.554159+0000","last_fullsized":"2024-03-30T13:18:44.554159+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:17:35.661077+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:17:35.661077+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:59.785888+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1615,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T13:18:46.571059+0000","last_change":"2024-03-30T13:18:14.814370+0000","last_active":"2024-03-30T13:18:46.571059+0000","last_peered":"2024-03-30T13:18:46.571059+0000","last_clean":"2024-03-30T13:18:46.571059+0000","last_became_active":"2024-03-30T13:18:14.277074+0000","last_became_peered":"2024-03-30T13:18:14.277074+0000","last_unstale":"2024-03-30T13:18:46.571059+0000","last_undegraded":"2024-03-30T13:18:46.571059+0000","last_fullsized":"2024-03-30T13:18:46.571059+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:18:14.814123+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:18:14.814123+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:32.802268+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":81210368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1221,"reported_epoch":640,"state":"active+clean","last_fresh":"2024-03-30T13:18:42.925650+0000","last_change":"2024-03-30T13:18:32.417767+0000","last_active":"2024-03-30T13:18:42.925650+0000","last_peered":"2024-03-30T13:18:42.925650+0000","last_clean":"2024-03-30T13:18:42.925650+0000","last_became_active":"2024-03-30T13:18:32.416688+0000","last_became_peered":"2024-03-30T13:18:32.416688+0000","last_unstale":"2024-03-30T13:18:42.925650+0000","last_undegraded":"2024-03-30T13:18:42.925650+0000","last_fullsized":"2024-03-30T13:18:42.925650+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:17:36.151638+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_clean_scrub_stamp":"2024-03-30T13:17:36.151638+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:03.568081+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1463,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-03-30T13:18:45.786301+0000","last_change":"2024-03-30T13:18:32.414752+0000","last_active":"2024-03-30T13:18:45.786301+0000","last_peered":"2024-03-30T13:18:45.786301+0000","last_clean":"2024-03-30T13:18:45.786301+0000","last_became_active":"2024-03-30T13:18:32.412897+0000","last_became_peered":"2024-03-30T13:18:32.412897+0000","last_unstale":"2024-03-30T13:18:45.786301+0000","last_undegraded":"2024-03-30T13:18:45.786301+0000","last_fullsized":"2024-03-30T13:18:45.786301+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_clean_scrub_stamp":"2024-03-30T13:17:50.531859+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:15.674072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1514,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-03-30T13:18:45.786267+0000","last_change":"2024-03-30T13:18:32.421100+0000","last_active":"2024-03-30T13:18:45.786267+0000","last_peered":"2024-03-30T13:18:45.786267+0000","last_clean":"2024-03-30T13:18:45.786267+0000","last_became_active":"2024-03-30T13:18:32.419902+0000","last_became_peered":"2024-03-30T13:18:32.419902+0000","last_unstale":"2024-03-30T13:18:45.786267+0000","last_undegraded":"2024-03-30T13:18:45.786267+0000","last_fullsized":"2024-03-30T13:18:45.786267+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:17:40.468714+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:17:40.468714+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:56.619526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":321,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T13:18:46.624490+0000","last_change":"2024-03-30T13:17:40.956701+0000","last_active":"2024-03-30T13:18:46.624490+0000","last_peered":"2024-03-30T13:18:46.624490+0000","last_clean":"2024-03-30T13:18:46.624490+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:18:46.624490+0000","last_undegraded":"2024-03-30T13:18:46.624490+0000","last_fullsized":"2024-03-30T13:18:46.624490+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:17:40.956450+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:17:40.956450+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:59.963567+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":124,"reported_epoch":641,"state":"active+clean","last_fresh":"2024-03-30T13:18:43.547897+0000","last_change":"2024-03-30T13:18:35.867309+0000","last_active":"2024-03-30T13:18:43.547897+0000","last_peered":"2024-03-30T13:18:43.547897+0000","last_clean":"2024-03-30T13:18:43.547897+0000","last_became_active":"2024-03-30T13:17:44.935038+0000","last_became_peered":"2024-03-30T13:17:44.935038+0000","last_unstale":"2024-03-30T13:18:43.547897+0000","last_undegraded":"2024-03-30T13:18:43.547897+0000","last_fullsized":"2024-03-30T13:18:43.547897+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:18:35.867018+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:18:35.867018+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:35.869590+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":327,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T13:18:46.624053+0000","last_change":"2024-03-30T13:18:32.416876+0000","last_active":"2024-03-30T13:18:46.624053+0000","last_peered":"2024-03-30T13:18:46.624053+0000","last_clean":"2024-03-30T13:18:46.624053+0000","last_became_active":"2024-03-30T13:18:32.415413+0000","last_became_peered":"2024-03-30T13:18:32.415413+0000","last_unstale":"2024-03-30T13:18:46.624053+0000","last_undegraded":"2024-03-30T13:18:46.624053+0000","last_fullsized":"2024-03-30T13:18:46.624053+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:18:10.804054+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:18:10.804054+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:16.441042+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":674,"reported_epoch":642,"state":"active+clean","last_fresh":"2024-03-30T13:18:44.554312+0000","last_change":"2024-03-30T13:17:58.851110+0000","last_active":"2024-03-30T13:18:44.554312+0000","last_peered":"2024-03-30T13:18:44.554312+0000","last_clean":"2024-03-30T13:18:44.554312+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:18:44.554312+0000","last_undegraded":"2024-03-30T13:18:44.554312+0000","last_fullsized":"2024-03-30T13:18:44.554312+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:17:58.850996+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:15.607294+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":230185570,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":629,"seq":2701534429189,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59528,"kb_used_data":22776,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94312312,"statfs":{"total":96636764160,"available":96575807488,"internally_reserved":0,"allocated":23322624,"data_stored":19012361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157492,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65184,"kb_used_data":27532,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94306656,"statfs":{"total":96636764160,"available":96570015744,"internally_reserved":0,"allocated":28192768,"data_stored":23875704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":559,"seq":2400886718495,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56832,"kb_used_data":19188,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94315008,"statfs":{"total":96636764160,"available":96578568192,"internally_reserved":0,"allocated":19648512,"data_stored":15358602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12639,"internal_metadata":38522529},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509722,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49308,"kb_used_data":13968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94322532,"statfs":{"total":96636764160,"available":96586272768,"internally_reserved":0,"allocated":14303232,"data_stored":10006571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375999001,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50456,"kb_used_data":13256,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94321384,"statfs":{"total":96636764160,"available":96585097216,"internally_reserved":0,"allocated":13574144,"data_stored":9270682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509723,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56480,"kb_used_data":21132,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315360,"statfs":{"total":96636764160,"available":96578928640,"internally_reserved":0,"allocated":21639168,"data_stored":17349258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":604,"seq":2594160246799,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55336,"kb_used_data":17688,"kb_used_omap":13,"kb_used_meta":37618,"kb_avail":94316504,"statfs":{"total":96636764160,"available":96580100096,"internally_reserved":0,"allocated":18112512,"data_stored":13815851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14282,"internal_metadata":38520886},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147432,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51884,"kb_used_data":20180,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94319956,"statfs":{"total":96636764160,"available":96583634944,"internally_reserved":0,"allocated":20664320,"data_stored":16347256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:18:49.756 INFO:tasks.thrashosds.thrasher:Choosing PG 1.0 with acting set [7, 0, 6] 2024-03-30T13:18:49.756 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 2, 3, 5, 4, 1, 7, 6] 2024-03-30T13:18:49.756 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-30T13:18:49.757 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:18:49.818 INFO:tasks.ceph.osd.7:Stopped 2024-03-30T13:18:49.818 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 2, 3, 5, 4, 1, 7, 6] 2024-03-30T13:18:49.818 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 7 2024-03-30T13:18:49.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:49.932+0000 7f9749fe6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:49.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:49.948+0000 7f9749fe6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:50.152 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:18:50.148+0000 7fbb38f08640 -1 mon.a@0(leader).osd e645 definitely_dead 0 2024-03-30T13:18:50.616 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.7. 2024-03-30T13:18:50.645 INFO:tasks.thrashosds.thrasher:dead_osds=[7], live_osds=[0, 2, 3, 5, 4, 1, 6] 2024-03-30T13:18:50.645 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:18:50.645 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:18:50.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:50.760+0000 7fd27a8c9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:50.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:18:50.772+0000 7fd27a8c9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:18:50.944 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:18:50.944 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:18:50.961 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1338,"stamp":"2024-03-30T13:18:50.621288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":230185570,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":389884,"kb_used_data":138236,"kb_used_omap":47,"kb_used_meta":251472,"kb_avail":660212996,"statfs":{"total":676457349120,"available":676058107904,"internally_reserved":0,"allocated":141553664,"data_stored":111313813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48264,"internal_metadata":257508216},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.064697"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1486,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-03-30T13:18:45.786479+0000","last_change":"2024-03-30T13:18:32.417584+0000","last_active":"2024-03-30T13:18:45.786479+0000","last_peered":"2024-03-30T13:18:45.786479+0000","last_clean":"2024-03-30T13:18:45.786479+0000","last_became_active":"2024-03-30T13:18:32.416374+0000","last_became_peered":"2024-03-30T13:18:32.416374+0000","last_unstale":"2024-03-30T13:18:45.786479+0000","last_undegraded":"2024-03-30T13:18:45.786479+0000","last_fullsized":"2024-03-30T13:18:45.786479+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:17:53.569165+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:56.559697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1376,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-03-30T13:18:47.590551+0000","last_change":"2024-03-30T13:18:33.123163+0000","last_active":"2024-03-30T13:18:47.590551+0000","last_peered":"2024-03-30T13:18:47.590551+0000","last_clean":"2024-03-30T13:18:47.590551+0000","last_became_active":"2024-03-30T13:17:44.937731+0000","last_became_peered":"2024-03-30T13:17:44.937731+0000","last_unstale":"2024-03-30T13:18:47.590551+0000","last_undegraded":"2024-03-30T13:18:47.590551+0000","last_fullsized":"2024-03-30T13:18:47.590551+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:18:33.122892+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:18:33.122892+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:37.289093+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":132,"reported_epoch":641,"state":"stale+active+clean","last_fresh":"2024-03-30T13:18:43.547905+0000","last_change":"2024-03-30T13:18:22.850213+0000","last_active":"2024-03-30T13:18:43.547905+0000","last_peered":"2024-03-30T13:18:43.547905+0000","last_clean":"2024-03-30T13:18:43.547905+0000","last_became_active":"2024-03-30T13:17:44.937300+0000","last_became_peered":"2024-03-30T13:17:44.937300+0000","last_unstale":"2024-03-30T13:18:50.621308+0000","last_undegraded":"2024-03-30T13:18:43.547905+0000","last_fullsized":"2024-03-30T13:18:43.547905+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:18:22.849961+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:18:22.849961+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:26.050541+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":138,"reported_epoch":641,"state":"stale+active+clean","last_fresh":"2024-03-30T13:18:43.547897+0000","last_change":"2024-03-30T13:18:32.422320+0000","last_active":"2024-03-30T13:18:43.547897+0000","last_peered":"2024-03-30T13:18:43.547897+0000","last_clean":"2024-03-30T13:18:43.547897+0000","last_became_active":"2024-03-30T13:18:32.421168+0000","last_became_peered":"2024-03-30T13:18:32.421168+0000","last_unstale":"2024-03-30T13:18:50.621310+0000","last_undegraded":"2024-03-30T13:18:43.547897+0000","last_fullsized":"2024-03-30T13:18:43.547897+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:17:46.411360+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:17:46.411360+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:54.421917+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":309,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T13:18:46.624266+0000","last_change":"2024-03-30T13:18:04.841148+0000","last_active":"2024-03-30T13:18:46.624266+0000","last_peered":"2024-03-30T13:18:46.624266+0000","last_clean":"2024-03-30T13:18:46.624266+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:18:46.624266+0000","last_undegraded":"2024-03-30T13:18:46.624266+0000","last_fullsized":"2024-03-30T13:18:46.624266+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:18:04.840923+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:18:04.840923+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:14.630038+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1477,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T13:18:46.571219+0000","last_change":"2024-03-30T13:18:42.729125+0000","last_active":"2024-03-30T13:18:46.571219+0000","last_peered":"2024-03-30T13:18:46.571219+0000","last_clean":"2024-03-30T13:18:46.571219+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:18:46.571219+0000","last_undegraded":"2024-03-30T13:18:46.571219+0000","last_fullsized":"2024-03-30T13:18:46.571219+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_clean_scrub_stamp":"2024-03-30T13:18:42.729016+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:03.586884+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":158,"reported_epoch":641,"state":"stale+active+clean","last_fresh":"2024-03-30T13:18:43.547897+0000","last_change":"2024-03-30T13:18:27.807956+0000","last_active":"2024-03-30T13:18:43.547897+0000","last_peered":"2024-03-30T13:18:43.547897+0000","last_clean":"2024-03-30T13:18:43.547897+0000","last_became_active":"2024-03-30T13:17:44.935396+0000","last_became_peered":"2024-03-30T13:17:44.935396+0000","last_unstale":"2024-03-30T13:18:50.621314+0000","last_undegraded":"2024-03-30T13:18:43.547897+0000","last_fullsized":"2024-03-30T13:18:43.547897+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:18:27.807804+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:18:27.807804+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:56.928075+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":719,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-03-30T13:18:47.635950+0000","last_change":"2024-03-30T13:18:32.414854+0000","last_active":"2024-03-30T13:18:47.635950+0000","last_peered":"2024-03-30T13:18:47.635950+0000","last_clean":"2024-03-30T13:18:47.635950+0000","last_became_active":"2024-03-30T13:18:32.413904+0000","last_became_peered":"2024-03-30T13:18:32.413904+0000","last_unstale":"2024-03-30T13:18:47.635950+0000","last_undegraded":"2024-03-30T13:18:47.635950+0000","last_fullsized":"2024-03-30T13:18:47.635950+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:17:35.661077+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:17:35.661077+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:59.785888+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1615,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T13:18:46.571059+0000","last_change":"2024-03-30T13:18:14.814370+0000","last_active":"2024-03-30T13:18:46.571059+0000","last_peered":"2024-03-30T13:18:46.571059+0000","last_clean":"2024-03-30T13:18:46.571059+0000","last_became_active":"2024-03-30T13:18:14.277074+0000","last_became_peered":"2024-03-30T13:18:14.277074+0000","last_unstale":"2024-03-30T13:18:46.571059+0000","last_undegraded":"2024-03-30T13:18:46.571059+0000","last_fullsized":"2024-03-30T13:18:46.571059+0000","mapping_epoch":624,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":625,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:18:14.814123+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:18:14.814123+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:32.802268+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":81210368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1230,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-03-30T13:18:47.590400+0000","last_change":"2024-03-30T13:18:32.417767+0000","last_active":"2024-03-30T13:18:47.590400+0000","last_peered":"2024-03-30T13:18:47.590400+0000","last_clean":"2024-03-30T13:18:47.590400+0000","last_became_active":"2024-03-30T13:18:32.416688+0000","last_became_peered":"2024-03-30T13:18:32.416688+0000","last_unstale":"2024-03-30T13:18:47.590400+0000","last_undegraded":"2024-03-30T13:18:47.590400+0000","last_fullsized":"2024-03-30T13:18:47.590400+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:17:36.151638+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_clean_scrub_stamp":"2024-03-30T13:17:36.151638+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:03.568081+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1463,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-03-30T13:18:45.786301+0000","last_change":"2024-03-30T13:18:32.414752+0000","last_active":"2024-03-30T13:18:45.786301+0000","last_peered":"2024-03-30T13:18:45.786301+0000","last_clean":"2024-03-30T13:18:45.786301+0000","last_became_active":"2024-03-30T13:18:32.412897+0000","last_became_peered":"2024-03-30T13:18:32.412897+0000","last_unstale":"2024-03-30T13:18:45.786301+0000","last_undegraded":"2024-03-30T13:18:45.786301+0000","last_fullsized":"2024-03-30T13:18:45.786301+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_clean_scrub_stamp":"2024-03-30T13:17:50.531859+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:15.674072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1514,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-03-30T13:18:45.786267+0000","last_change":"2024-03-30T13:18:32.421100+0000","last_active":"2024-03-30T13:18:45.786267+0000","last_peered":"2024-03-30T13:18:45.786267+0000","last_clean":"2024-03-30T13:18:45.786267+0000","last_became_active":"2024-03-30T13:18:32.419902+0000","last_became_peered":"2024-03-30T13:18:32.419902+0000","last_unstale":"2024-03-30T13:18:45.786267+0000","last_undegraded":"2024-03-30T13:18:45.786267+0000","last_fullsized":"2024-03-30T13:18:45.786267+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:17:40.468714+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:17:40.468714+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:56.619526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":321,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T13:18:46.624490+0000","last_change":"2024-03-30T13:17:40.956701+0000","last_active":"2024-03-30T13:18:46.624490+0000","last_peered":"2024-03-30T13:18:46.624490+0000","last_clean":"2024-03-30T13:18:46.624490+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:18:46.624490+0000","last_undegraded":"2024-03-30T13:18:46.624490+0000","last_fullsized":"2024-03-30T13:18:46.624490+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:17:40.956450+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:17:40.956450+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:18:59.963567+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":124,"reported_epoch":641,"state":"stale+active+clean","last_fresh":"2024-03-30T13:18:43.547897+0000","last_change":"2024-03-30T13:18:35.867309+0000","last_active":"2024-03-30T13:18:43.547897+0000","last_peered":"2024-03-30T13:18:43.547897+0000","last_clean":"2024-03-30T13:18:43.547897+0000","last_became_active":"2024-03-30T13:17:44.935038+0000","last_became_peered":"2024-03-30T13:17:44.935038+0000","last_unstale":"2024-03-30T13:18:50.621317+0000","last_undegraded":"2024-03-30T13:18:43.547897+0000","last_fullsized":"2024-03-30T13:18:43.547897+0000","mapping_epoch":608,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":609,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:18:35.867018+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:18:35.867018+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:35.869590+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":327,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-03-30T13:18:46.624053+0000","last_change":"2024-03-30T13:18:32.416876+0000","last_active":"2024-03-30T13:18:46.624053+0000","last_peered":"2024-03-30T13:18:46.624053+0000","last_clean":"2024-03-30T13:18:46.624053+0000","last_became_active":"2024-03-30T13:18:32.415413+0000","last_became_peered":"2024-03-30T13:18:32.415413+0000","last_unstale":"2024-03-30T13:18:46.624053+0000","last_undegraded":"2024-03-30T13:18:46.624053+0000","last_fullsized":"2024-03-30T13:18:46.624053+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:18:10.804054+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:18:10.804054+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:16.441042+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":680,"reported_epoch":645,"state":"active+clean","last_fresh":"2024-03-30T13:18:47.635940+0000","last_change":"2024-03-30T13:17:58.851110+0000","last_active":"2024-03-30T13:18:47.635940+0000","last_peered":"2024-03-30T13:18:47.635940+0000","last_clean":"2024-03-30T13:18:47.635940+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:18:47.635940+0000","last_undegraded":"2024-03-30T13:18:47.635940+0000","last_fullsized":"2024-03-30T13:18:47.635940+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:17:58.850996+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:15.607294+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":230185570,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":629,"seq":2701534429189,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59528,"kb_used_data":22776,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94312312,"statfs":{"total":96636764160,"available":96575807488,"internally_reserved":0,"allocated":23322624,"data_stored":19012361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157492,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65184,"kb_used_data":27532,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94306656,"statfs":{"total":96636764160,"available":96570015744,"internally_reserved":0,"allocated":28192768,"data_stored":23875704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":559,"seq":2400886718496,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56900,"kb_used_data":19248,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94314940,"statfs":{"total":96636764160,"available":96578498560,"internally_reserved":0,"allocated":19709952,"data_stored":15386087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12639,"internal_metadata":38522529},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509723,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49360,"kb_used_data":14016,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94322480,"statfs":{"total":96636764160,"available":96586219520,"internally_reserved":0,"allocated":14352384,"data_stored":10028519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375999002,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50492,"kb_used_data":13292,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94321348,"statfs":{"total":96636764160,"available":96585060352,"internally_reserved":0,"allocated":13611008,"data_stored":9287143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509724,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56536,"kb_used_data":21192,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315304,"statfs":{"total":96636764160,"available":96578871296,"internally_reserved":0,"allocated":21700608,"data_stored":17376743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147432,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51884,"kb_used_data":20180,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94319956,"statfs":{"total":96636764160,"available":96583634944,"internally_reserved":0,"allocated":20664320,"data_stored":16347256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:18:50.962 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:19:15.963 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:19:15.964 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:19:16.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:16.087+0000 7fdd15545640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:16.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:16.103+0000 7fdd15545640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:16.279 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:19:16.280 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:19:16.296 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1352,"stamp":"2024-03-30T13:19:16.194927+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":262857949,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":400072,"kb_used_data":148496,"kb_used_omap":47,"kb_used_meta":251472,"kb_avail":660202808,"statfs":{"total":676457349120,"available":676047675392,"internally_reserved":0,"allocated":152059904,"data_stored":121699147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48264,"internal_metadata":257508216},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004069"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1502,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-03-30T13:18:57.255298+0000","last_change":"2024-03-30T13:18:57.255371+0000","last_active":"2024-03-30T13:18:57.255298+0000","last_peered":"2024-03-30T13:18:57.255298+0000","last_clean":"2024-03-30T13:18:57.255298+0000","last_became_active":"2024-03-30T13:18:32.416374+0000","last_became_peered":"2024-03-30T13:18:32.416374+0000","last_unstale":"2024-03-30T13:18:57.255298+0000","last_undegraded":"2024-03-30T13:18:57.255298+0000","last_fullsized":"2024-03-30T13:18:57.255298+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:18:57.255102+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:18:57.255102+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:01.922986+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1390,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-03-30T13:18:51.677932+0000","last_change":"2024-03-30T13:18:51.677932+0000","last_active":"2024-03-30T13:18:51.677932+0000","last_peered":"2024-03-30T13:18:51.677932+0000","last_clean":"2024-03-30T13:18:51.677932+0000","last_became_active":"2024-03-30T13:18:51.644102+0000","last_became_peered":"2024-03-30T13:18:51.644102+0000","last_unstale":"2024-03-30T13:18:51.677932+0000","last_undegraded":"2024-03-30T13:18:51.677932+0000","last_fullsized":"2024-03-30T13:18:51.677932+0000","mapping_epoch":646,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":647,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:18:33.122892+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:18:33.122892+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:33.508914+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10621892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":19,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-03-30T13:18:51.736372+0000","last_change":"2024-03-30T13:18:51.735334+0000","last_active":"2024-03-30T13:18:51.736372+0000","last_peered":"2024-03-30T13:18:51.736372+0000","last_clean":"2024-03-30T13:18:51.736372+0000","last_became_active":"2024-03-30T13:18:51.635318+0000","last_became_peered":"2024-03-30T13:18:51.635318+0000","last_unstale":"2024-03-30T13:18:51.736372+0000","last_undegraded":"2024-03-30T13:18:51.736372+0000","last_fullsized":"2024-03-30T13:18:51.736372+0000","mapping_epoch":646,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":647,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:18:22.849961+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:18:22.849961+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:33.267732+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":25,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-03-30T13:18:58.851295+0000","last_change":"2024-03-30T13:18:58.851365+0000","last_active":"2024-03-30T13:18:58.851295+0000","last_peered":"2024-03-30T13:18:58.851295+0000","last_clean":"2024-03-30T13:18:58.851295+0000","last_became_active":"2024-03-30T13:18:51.635853+0000","last_became_peered":"2024-03-30T13:18:51.635853+0000","last_unstale":"2024-03-30T13:18:58.851295+0000","last_undegraded":"2024-03-30T13:18:58.851295+0000","last_fullsized":"2024-03-30T13:18:58.851295+0000","mapping_epoch":646,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":647,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:18:58.851088+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:18:58.851088+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:11.438616+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":315,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-03-30T13:18:51.632256+0000","last_change":"2024-03-30T13:18:04.841148+0000","last_active":"2024-03-30T13:18:51.632256+0000","last_peered":"2024-03-30T13:18:51.632256+0000","last_clean":"2024-03-30T13:18:51.632256+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:18:51.632256+0000","last_undegraded":"2024-03-30T13:18:51.632256+0000","last_fullsized":"2024-03-30T13:18:51.632256+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:18:04.840923+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:18:04.840923+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:14.630038+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1483,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-03-30T13:18:51.639130+0000","last_change":"2024-03-30T13:18:42.729125+0000","last_active":"2024-03-30T13:18:51.639130+0000","last_peered":"2024-03-30T13:18:51.639130+0000","last_clean":"2024-03-30T13:18:51.639130+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:18:51.639130+0000","last_undegraded":"2024-03-30T13:18:51.639130+0000","last_fullsized":"2024-03-30T13:18:51.639130+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_clean_scrub_stamp":"2024-03-30T13:18:42.729016+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:03.586884+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":32,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-03-30T13:18:51.791415+0000","last_change":"2024-03-30T13:18:51.790433+0000","last_active":"2024-03-30T13:18:51.791415+0000","last_peered":"2024-03-30T13:18:51.791415+0000","last_clean":"2024-03-30T13:18:51.791415+0000","last_became_active":"2024-03-30T13:18:51.646797+0000","last_became_peered":"2024-03-30T13:18:51.646797+0000","last_unstale":"2024-03-30T13:18:51.791415+0000","last_undegraded":"2024-03-30T13:18:51.791415+0000","last_fullsized":"2024-03-30T13:18:51.791415+0000","mapping_epoch":646,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":647,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:18:27.807804+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:18:27.807804+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:28.743278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":732,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-03-30T13:18:59.894473+0000","last_change":"2024-03-30T13:18:59.894546+0000","last_active":"2024-03-30T13:18:59.894473+0000","last_peered":"2024-03-30T13:18:59.894473+0000","last_clean":"2024-03-30T13:18:59.894473+0000","last_became_active":"2024-03-30T13:18:32.413904+0000","last_became_peered":"2024-03-30T13:18:32.413904+0000","last_unstale":"2024-03-30T13:18:59.894473+0000","last_undegraded":"2024-03-30T13:18:59.894473+0000","last_fullsized":"2024-03-30T13:18:59.894473+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:18:59.894294+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:18:59.894294+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:21.905125+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1633,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-03-30T13:18:51.735179+0000","last_change":"2024-03-30T13:18:51.735179+0000","last_active":"2024-03-30T13:18:51.735179+0000","last_peered":"2024-03-30T13:18:51.735179+0000","last_clean":"2024-03-30T13:18:51.735179+0000","last_became_active":"2024-03-30T13:18:51.642295+0000","last_became_peered":"2024-03-30T13:18:51.642295+0000","last_unstale":"2024-03-30T13:18:51.735179+0000","last_undegraded":"2024-03-30T13:18:51.735179+0000","last_fullsized":"2024-03-30T13:18:51.735179+0000","mapping_epoch":646,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":647,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:18:14.814123+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:18:14.814123+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:28.540268+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":86286016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1242,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-03-30T13:19:03.699043+0000","last_change":"2024-03-30T13:19:03.699231+0000","last_active":"2024-03-30T13:19:03.699043+0000","last_peered":"2024-03-30T13:19:03.699043+0000","last_clean":"2024-03-30T13:19:03.699043+0000","last_became_active":"2024-03-30T13:18:32.416688+0000","last_became_peered":"2024-03-30T13:18:32.416688+0000","last_unstale":"2024-03-30T13:19:03.699043+0000","last_undegraded":"2024-03-30T13:19:03.699043+0000","last_fullsized":"2024-03-30T13:19:03.699043+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:19:03.698837+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_clean_scrub_stamp":"2024-03-30T13:19:03.698837+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:21.113435+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1471,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-03-30T13:18:51.625710+0000","last_change":"2024-03-30T13:18:32.414752+0000","last_active":"2024-03-30T13:18:51.625710+0000","last_peered":"2024-03-30T13:18:51.625710+0000","last_clean":"2024-03-30T13:18:51.625710+0000","last_became_active":"2024-03-30T13:18:32.412897+0000","last_became_peered":"2024-03-30T13:18:32.412897+0000","last_unstale":"2024-03-30T13:18:51.625710+0000","last_undegraded":"2024-03-30T13:18:51.625710+0000","last_fullsized":"2024-03-30T13:18:51.625710+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_clean_scrub_stamp":"2024-03-30T13:17:50.531859+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:15.674072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1530,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-03-30T13:19:05.199070+0000","last_change":"2024-03-30T13:19:05.199227+0000","last_active":"2024-03-30T13:19:05.199070+0000","last_peered":"2024-03-30T13:19:05.199070+0000","last_clean":"2024-03-30T13:19:05.199070+0000","last_became_active":"2024-03-30T13:18:32.419902+0000","last_became_peered":"2024-03-30T13:18:32.419902+0000","last_unstale":"2024-03-30T13:19:05.199070+0000","last_undegraded":"2024-03-30T13:19:05.199070+0000","last_fullsized":"2024-03-30T13:19:05.199070+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:19:05.198795+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:19:05.198795+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:32.312402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":336,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-03-30T13:19:00.934348+0000","last_change":"2024-03-30T13:19:00.934425+0000","last_active":"2024-03-30T13:19:00.934348+0000","last_peered":"2024-03-30T13:19:00.934348+0000","last_clean":"2024-03-30T13:19:00.934348+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:19:00.934348+0000","last_undegraded":"2024-03-30T13:19:00.934348+0000","last_fullsized":"2024-03-30T13:19:00.934348+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:19:00.934164+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:19:00.934164+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:24.127933+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":15,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-03-30T13:18:51.674818+0000","last_change":"2024-03-30T13:18:51.673935+0000","last_active":"2024-03-30T13:18:51.674818+0000","last_peered":"2024-03-30T13:18:51.674818+0000","last_clean":"2024-03-30T13:18:51.674818+0000","last_became_active":"2024-03-30T13:18:51.642395+0000","last_became_peered":"2024-03-30T13:18:51.642395+0000","last_unstale":"2024-03-30T13:18:51.674818+0000","last_undegraded":"2024-03-30T13:18:51.674818+0000","last_fullsized":"2024-03-30T13:18:51.674818+0000","mapping_epoch":646,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":647,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:18:35.867018+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:18:35.867018+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:55.017774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":334,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-03-30T13:18:51.630621+0000","last_change":"2024-03-30T13:18:32.416876+0000","last_active":"2024-03-30T13:18:51.630621+0000","last_peered":"2024-03-30T13:18:51.630621+0000","last_clean":"2024-03-30T13:18:51.630621+0000","last_became_active":"2024-03-30T13:18:32.415413+0000","last_became_peered":"2024-03-30T13:18:32.415413+0000","last_unstale":"2024-03-30T13:18:51.630621+0000","last_undegraded":"2024-03-30T13:18:51.630621+0000","last_fullsized":"2024-03-30T13:18:51.630621+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:18:10.804054+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:18:10.804054+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:16.441042+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":685,"reported_epoch":647,"state":"active+clean","last_fresh":"2024-03-30T13:18:51.627041+0000","last_change":"2024-03-30T13:17:58.851110+0000","last_active":"2024-03-30T13:18:51.627041+0000","last_peered":"2024-03-30T13:18:51.627041+0000","last_clean":"2024-03-30T13:18:51.627041+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:18:51.627041+0000","last_undegraded":"2024-03-30T13:18:51.627041+0000","last_fullsized":"2024-03-30T13:18:51.627041+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:17:58.850996+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:15.607294+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":262857949,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":107036672,"data_stored":107036672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":629,"seq":2701534429194,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59568,"kb_used_data":22824,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94312272,"statfs":{"total":96636764160,"available":96575766528,"internally_reserved":0,"allocated":23371776,"data_stored":19034525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157497,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65848,"kb_used_data":28208,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94305992,"statfs":{"total":96636764160,"available":96569335808,"internally_reserved":0,"allocated":28884992,"data_stored":24547741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":559,"seq":2400886718501,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56912,"kb_used_data":19272,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94314928,"statfs":{"total":96636764160,"available":96578486272,"internally_reserved":0,"allocated":19734528,"data_stored":15397277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12639,"internal_metadata":38522529},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509728,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55472,"kb_used_data":20136,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94316368,"statfs":{"total":96636764160,"available":96579960832,"internally_reserved":0,"allocated":20619264,"data_stored":16282013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375999007,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53816,"kb_used_data":16624,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94318024,"statfs":{"total":96636764160,"available":96581656576,"internally_reserved":0,"allocated":17022976,"data_stored":12685725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509729,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56552,"kb_used_data":21216,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315288,"statfs":{"total":96636764160,"available":96578854912,"internally_reserved":0,"allocated":21725184,"data_stored":17387933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147437,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51904,"kb_used_data":20216,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94319936,"statfs":{"total":96636764160,"available":96583614464,"internally_reserved":0,"allocated":20701184,"data_stored":16363933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9121792,"data_stored":9121792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12718080,"data_stored":12718080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20983808,"data_stored":20983808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:19:16.297 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:19:16.297 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:19:16.297 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2024-03-30T13:19:16.297 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-30T13:19:16.297 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-30T13:19:16.300 INFO:tasks.ceph.osd.7:Started 2024-03-30T13:19:16.300 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-30T13:19:16.363 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:19:16.363+0000 7fad3b5017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:16.370 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:19:16.367+0000 7fad3b5017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:16.370 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:19:16.367+0000 7fad3b5017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:16.421 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T13:19:16.421 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T13:19:16.421 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T13:19:16.421 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T13:19:16.421 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T13:19:16.422 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T13:19:16.422 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-03-30T13:19:16.422 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-03-30T13:19:16.422 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-03-30T13:19:16.422 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-03-30T13:19:16.422 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-03-30T13:19:16.422 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T13:19:16.424 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:19:16.424 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-03-30T13:19:16.912 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:19:16.911+0000 7fad3b5017c0 -1 Falling back to public interface 2024-03-30T13:19:20.696 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:19:20.695+0000 7fad3b5017c0 -1 osd.7 645 log_to_monitors true 2024-03-30T13:19:21.396 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:19:21.395+0000 7fad332d6640 -1 osd.7 645 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:19:21.425 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-30T13:19:21.558 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T13:19:21.559 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T13:19:21.559 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T13:19:21.559 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T13:19:21.572 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:19:21.572 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-03-30T13:19:21.572 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 7 2024-03-30T13:19:21.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:21.699+0000 7fe6840a4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:21.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:21.711+0000 7fe6840a4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:21.901 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:19:21.899+0000 7fbb38f08640 -1 mon.a@0(leader).osd e649 definitely_dead 0 2024-03-30T13:19:22.363 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.7. 2024-03-30T13:19:22.391 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-03-30T13:19:22.391 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:19:22.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:22.503+0000 7f238a377640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:22.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:22.519+0000 7f238a377640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:23.366 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:19:23.401 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:19:23.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:23.515+0000 7f3de2b54640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:23.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:23.531+0000 7f3de2b54640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:24.388 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:19:24.412 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:19:24.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:24.531+0000 7f7240bdc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:24.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:24.547+0000 7f7240bdc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:25.392 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:19:25.417 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:19:25.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:25.531+0000 7f98bbada640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:25.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:25.547+0000 7f98bbada640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:26.417 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:19:26.447 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:19:26.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:26.563+0000 7fd5ed694640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:26.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:26.579+0000 7fd5ed694640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:27.413 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:19:27.438 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:19:27.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:27.555+0000 7f9e304ea640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:27.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:27.571+0000 7f9e304ea640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:28.429 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:19:28.453 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:19:28.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:28.567+0000 7f8a6fb0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:28.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:28.583+0000 7f8a6fb0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:29.441 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:19:29.467 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:19:29.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:29.583+0000 7fefd9291640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:29.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:29.599+0000 7fefd9291640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:30.445 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:19:30.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:19:30.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:30.583+0000 7f9495821640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:30.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:30.599+0000 7f9495821640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:31.457 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:19:31.485 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:19:31.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:31.603+0000 7fd82e530640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:31.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:31.619+0000 7fd82e530640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:32.474 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:19:32.498 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:19:32.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:32.615+0000 7f3bb74c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:32.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:32.631+0000 7f3bb74c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:33.485 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:19:33.514 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:19:33.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:33.631+0000 7f36ad436640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:33.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:33.647+0000 7f36ad436640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:34.490 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:19:34.515 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:19:34.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:34.631+0000 7f86cbd56640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:34.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:34.647+0000 7f86cbd56640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:35.500 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:19:35.527 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:19:35.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:35.647+0000 7fd959ca0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:35.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:35.663+0000 7fd959ca0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:36.516 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:19:36.542 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:19:36.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:36.659+0000 7fd65e097640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:36.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:36.675+0000 7fd65e097640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:37.529 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:19:37.553 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:19:37.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:37.667+0000 7f5a34a2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:37.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:37.683+0000 7f5a34a2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:38.543 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:19:38.568 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:19:38.568 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:19:38.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:38.679+0000 7f3547982640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:38.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:38.695+0000 7f3547982640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:38.868 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:19:38.868 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:19:38.886 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1382,"stamp":"2024-03-30T13:19:38.548169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":247967145,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":445740,"kb_used_data":157676,"kb_used_omap":62,"kb_used_meta":287873,"kb_avail":754528980,"statfs":{"total":773094113280,"available":772637675520,"internally_reserved":0,"allocated":161460224,"data_stored":125945702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64197,"internal_metadata":294782267},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.053019"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1534,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T13:19:35.778196+0000","last_change":"2024-03-30T13:18:57.255371+0000","last_active":"2024-03-30T13:19:35.778196+0000","last_peered":"2024-03-30T13:19:35.778196+0000","last_clean":"2024-03-30T13:19:35.778196+0000","last_became_active":"2024-03-30T13:18:32.416374+0000","last_became_peered":"2024-03-30T13:18:32.416374+0000","last_unstale":"2024-03-30T13:19:35.778196+0000","last_undegraded":"2024-03-30T13:19:35.778196+0000","last_fullsized":"2024-03-30T13:19:35.778196+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:18:57.255102+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:18:57.255102+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:01.922986+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1424,"reported_epoch":661,"state":"active+clean","last_fresh":"2024-03-30T13:19:33.487311+0000","last_change":"2024-03-30T13:19:23.389925+0000","last_active":"2024-03-30T13:19:33.487311+0000","last_peered":"2024-03-30T13:19:33.487311+0000","last_clean":"2024-03-30T13:19:33.487311+0000","last_became_active":"2024-03-30T13:19:23.388425+0000","last_became_peered":"2024-03-30T13:19:23.388425+0000","last_unstale":"2024-03-30T13:19:33.487311+0000","last_undegraded":"2024-03-30T13:19:33.487311+0000","last_fullsized":"2024-03-30T13:19:33.487311+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:18:33.122892+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:18:33.122892+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:54.688629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10621892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":149,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:37.233852+0000","last_change":"2024-03-30T13:19:30.741906+0000","last_active":"2024-03-30T13:19:37.233852+0000","last_peered":"2024-03-30T13:19:37.233852+0000","last_clean":"2024-03-30T13:19:37.233852+0000","last_became_active":"2024-03-30T13:19:23.383364+0000","last_became_peered":"2024-03-30T13:19:23.383364+0000","last_unstale":"2024-03-30T13:19:37.233852+0000","last_undegraded":"2024-03-30T13:19:37.233852+0000","last_fullsized":"2024-03-30T13:19:37.233852+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:19:30.741663+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:19:30.741663+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:56.023911+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":156,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:37.233887+0000","last_change":"2024-03-30T13:19:23.950933+0000","last_active":"2024-03-30T13:19:37.233887+0000","last_peered":"2024-03-30T13:19:37.233887+0000","last_clean":"2024-03-30T13:19:37.233887+0000","last_became_active":"2024-03-30T13:19:23.949921+0000","last_became_peered":"2024-03-30T13:19:23.949921+0000","last_unstale":"2024-03-30T13:19:37.233887+0000","last_undegraded":"2024-03-30T13:19:37.233887+0000","last_fullsized":"2024-03-30T13:19:37.233887+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:18:58.851088+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:18:58.851088+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:07.550478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":354,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:37.233664+0000","last_change":"2024-03-30T13:19:16.856148+0000","last_active":"2024-03-30T13:19:37.233664+0000","last_peered":"2024-03-30T13:19:37.233664+0000","last_clean":"2024-03-30T13:19:37.233664+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:19:37.233664+0000","last_undegraded":"2024-03-30T13:19:37.233664+0000","last_fullsized":"2024-03-30T13:19:37.233664+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:19:16.855901+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:19:16.855901+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:19.845199+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1516,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:36.567998+0000","last_change":"2024-03-30T13:18:42.729125+0000","last_active":"2024-03-30T13:19:36.567998+0000","last_peered":"2024-03-30T13:19:36.567998+0000","last_clean":"2024-03-30T13:19:36.567998+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:19:36.567998+0000","last_undegraded":"2024-03-30T13:19:36.567998+0000","last_fullsized":"2024-03-30T13:19:36.567998+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_clean_scrub_stamp":"2024-03-30T13:18:42.729016+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:03.586884+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":191,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:37.234020+0000","last_change":"2024-03-30T13:19:35.780363+0000","last_active":"2024-03-30T13:19:37.234020+0000","last_peered":"2024-03-30T13:19:37.234020+0000","last_clean":"2024-03-30T13:19:37.234020+0000","last_became_active":"2024-03-30T13:19:23.388140+0000","last_became_peered":"2024-03-30T13:19:23.388140+0000","last_unstale":"2024-03-30T13:19:37.234020+0000","last_undegraded":"2024-03-30T13:19:37.234020+0000","last_fullsized":"2024-03-30T13:19:37.234020+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:19:35.780073+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:19:35.780073+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:59.577918+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":761,"reported_epoch":662,"state":"active+clean","last_fresh":"2024-03-30T13:19:34.495472+0000","last_change":"2024-03-30T13:18:59.894546+0000","last_active":"2024-03-30T13:19:34.495472+0000","last_peered":"2024-03-30T13:19:34.495472+0000","last_clean":"2024-03-30T13:19:34.495472+0000","last_became_active":"2024-03-30T13:18:32.413904+0000","last_became_peered":"2024-03-30T13:18:32.413904+0000","last_unstale":"2024-03-30T13:19:34.495472+0000","last_undegraded":"2024-03-30T13:19:34.495472+0000","last_fullsized":"2024-03-30T13:19:34.495472+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:18:59.894294+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:18:59.894294+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:21.905125+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1671,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:36.567961+0000","last_change":"2024-03-30T13:19:23.380909+0000","last_active":"2024-03-30T13:19:36.567961+0000","last_peered":"2024-03-30T13:19:36.567961+0000","last_clean":"2024-03-30T13:19:36.567961+0000","last_became_active":"2024-03-30T13:19:23.380536+0000","last_became_peered":"2024-03-30T13:19:23.380536+0000","last_unstale":"2024-03-30T13:19:36.567961+0000","last_undegraded":"2024-03-30T13:19:36.567961+0000","last_fullsized":"2024-03-30T13:19:36.567961+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:18:14.814123+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:18:14.814123+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":86286016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1270,"reported_epoch":661,"state":"active+clean","last_fresh":"2024-03-30T13:19:33.487230+0000","last_change":"2024-03-30T13:19:03.699231+0000","last_active":"2024-03-30T13:19:33.487230+0000","last_peered":"2024-03-30T13:19:33.487230+0000","last_clean":"2024-03-30T13:19:33.487230+0000","last_became_active":"2024-03-30T13:18:32.416688+0000","last_became_peered":"2024-03-30T13:18:32.416688+0000","last_unstale":"2024-03-30T13:19:33.487230+0000","last_undegraded":"2024-03-30T13:19:33.487230+0000","last_fullsized":"2024-03-30T13:19:33.487230+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:19:03.698837+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_clean_scrub_stamp":"2024-03-30T13:19:03.698837+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:21.113435+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1511,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T13:19:35.777871+0000","last_change":"2024-03-30T13:19:19.392352+0000","last_active":"2024-03-30T13:19:35.777871+0000","last_peered":"2024-03-30T13:19:35.777871+0000","last_clean":"2024-03-30T13:19:35.777871+0000","last_became_active":"2024-03-30T13:18:32.412897+0000","last_became_peered":"2024-03-30T13:18:32.412897+0000","last_unstale":"2024-03-30T13:19:35.777871+0000","last_undegraded":"2024-03-30T13:19:35.777871+0000","last_fullsized":"2024-03-30T13:19:35.777871+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:19:19.392102+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_clean_scrub_stamp":"2024-03-30T13:19:19.392102+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:22.733398+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1562,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T13:19:35.778090+0000","last_change":"2024-03-30T13:19:05.199227+0000","last_active":"2024-03-30T13:19:35.778090+0000","last_peered":"2024-03-30T13:19:35.778090+0000","last_clean":"2024-03-30T13:19:35.778090+0000","last_became_active":"2024-03-30T13:18:32.419902+0000","last_became_peered":"2024-03-30T13:18:32.419902+0000","last_unstale":"2024-03-30T13:19:35.778090+0000","last_undegraded":"2024-03-30T13:19:35.778090+0000","last_fullsized":"2024-03-30T13:19:35.778090+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:19:05.198795+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:19:05.198795+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:32.312402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":367,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:37.233650+0000","last_change":"2024-03-30T13:19:00.934425+0000","last_active":"2024-03-30T13:19:37.233650+0000","last_peered":"2024-03-30T13:19:37.233650+0000","last_clean":"2024-03-30T13:19:37.233650+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:19:37.233650+0000","last_undegraded":"2024-03-30T13:19:37.233650+0000","last_fullsized":"2024-03-30T13:19:37.233650+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:19:00.934164+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:19:00.934164+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:24.127933+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":147,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:37.234130+0000","last_change":"2024-03-30T13:19:23.382202+0000","last_active":"2024-03-30T13:19:37.234130+0000","last_peered":"2024-03-30T13:19:37.234130+0000","last_clean":"2024-03-30T13:19:37.234130+0000","last_became_active":"2024-03-30T13:19:23.380857+0000","last_became_peered":"2024-03-30T13:19:23.380857+0000","last_unstale":"2024-03-30T13:19:37.234130+0000","last_undegraded":"2024-03-30T13:19:37.234130+0000","last_fullsized":"2024-03-30T13:19:37.234130+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:18:35.867018+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:18:35.867018+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:45.320201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":373,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:37.233661+0000","last_change":"2024-03-30T13:19:33.837451+0000","last_active":"2024-03-30T13:19:37.233661+0000","last_peered":"2024-03-30T13:19:37.233661+0000","last_clean":"2024-03-30T13:19:37.233661+0000","last_became_active":"2024-03-30T13:18:32.415413+0000","last_became_peered":"2024-03-30T13:18:32.415413+0000","last_unstale":"2024-03-30T13:19:37.233661+0000","last_undegraded":"2024-03-30T13:19:37.233661+0000","last_fullsized":"2024-03-30T13:19:37.233661+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:19:33.837082+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:19:33.837082+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:54.376722+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":722,"reported_epoch":662,"state":"active+clean","last_fresh":"2024-03-30T13:19:34.495751+0000","last_change":"2024-03-30T13:19:19.587739+0000","last_active":"2024-03-30T13:19:34.495751+0000","last_peered":"2024-03-30T13:19:34.495751+0000","last_clean":"2024-03-30T13:19:34.495751+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:19:34.495751+0000","last_undegraded":"2024-03-30T13:19:34.495751+0000","last_fullsized":"2024-03-30T13:19:34.495751+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:19:19.587471+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:19:19.587471+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:40.749837+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":247967145,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":629,"seq":2701534429199,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59772,"kb_used_data":23016,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94312068,"statfs":{"total":96636764160,"available":96575557632,"internally_reserved":0,"allocated":23568384,"data_stored":19123968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157502,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65408,"kb_used_data":27760,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94306432,"statfs":{"total":96636764160,"available":96569786368,"internally_reserved":0,"allocated":28426240,"data_stored":23981824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":559,"seq":2400886718505,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57084,"kb_used_data":19440,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94314756,"statfs":{"total":96636764160,"available":96578310144,"internally_reserved":0,"allocated":19906560,"data_stored":15475746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12639,"internal_metadata":38522529},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509732,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49556,"kb_used_data":14208,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94322284,"statfs":{"total":96636764160,"available":96586018816,"internally_reserved":0,"allocated":14548992,"data_stored":10118178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375999011,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50692,"kb_used_data":13496,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94321148,"statfs":{"total":96636764160,"available":96584855552,"internally_reserved":0,"allocated":13819904,"data_stored":9382289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509733,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56732,"kb_used_data":21384,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315108,"statfs":{"total":96636764160,"available":96578670592,"internally_reserved":0,"allocated":21897216,"data_stored":17466402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742405,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54380,"kb_used_data":17952,"kb_used_omap":15,"kb_used_meta":36400,"kb_avail":94317460,"statfs":{"total":96636764160,"available":96581079040,"internally_reserved":0,"allocated":18382848,"data_stored":13938432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":37274051},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147442,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52116,"kb_used_data":20420,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94319724,"statfs":{"total":96636764160,"available":96583397376,"internally_reserved":0,"allocated":20910080,"data_stored":16458863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:19:38.887 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:19:39.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:39.003+0000 7fe38d0ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:39.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:39.015+0000 7fe38d0ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:39.188 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:19:39.189 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:19:39.205 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1382,"stamp":"2024-03-30T13:19:38.548169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":247967145,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":445740,"kb_used_data":157676,"kb_used_omap":62,"kb_used_meta":287873,"kb_avail":754528980,"statfs":{"total":773094113280,"available":772637675520,"internally_reserved":0,"allocated":161460224,"data_stored":125945702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64197,"internal_metadata":294782267},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.053019"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1534,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T13:19:35.778196+0000","last_change":"2024-03-30T13:18:57.255371+0000","last_active":"2024-03-30T13:19:35.778196+0000","last_peered":"2024-03-30T13:19:35.778196+0000","last_clean":"2024-03-30T13:19:35.778196+0000","last_became_active":"2024-03-30T13:18:32.416374+0000","last_became_peered":"2024-03-30T13:18:32.416374+0000","last_unstale":"2024-03-30T13:19:35.778196+0000","last_undegraded":"2024-03-30T13:19:35.778196+0000","last_fullsized":"2024-03-30T13:19:35.778196+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:18:57.255102+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:18:57.255102+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:01.922986+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1424,"reported_epoch":661,"state":"active+clean","last_fresh":"2024-03-30T13:19:33.487311+0000","last_change":"2024-03-30T13:19:23.389925+0000","last_active":"2024-03-30T13:19:33.487311+0000","last_peered":"2024-03-30T13:19:33.487311+0000","last_clean":"2024-03-30T13:19:33.487311+0000","last_became_active":"2024-03-30T13:19:23.388425+0000","last_became_peered":"2024-03-30T13:19:23.388425+0000","last_unstale":"2024-03-30T13:19:33.487311+0000","last_undegraded":"2024-03-30T13:19:33.487311+0000","last_fullsized":"2024-03-30T13:19:33.487311+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:18:33.122892+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:18:33.122892+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:54.688629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10621892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":149,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:37.233852+0000","last_change":"2024-03-30T13:19:30.741906+0000","last_active":"2024-03-30T13:19:37.233852+0000","last_peered":"2024-03-30T13:19:37.233852+0000","last_clean":"2024-03-30T13:19:37.233852+0000","last_became_active":"2024-03-30T13:19:23.383364+0000","last_became_peered":"2024-03-30T13:19:23.383364+0000","last_unstale":"2024-03-30T13:19:37.233852+0000","last_undegraded":"2024-03-30T13:19:37.233852+0000","last_fullsized":"2024-03-30T13:19:37.233852+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:19:30.741663+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:19:30.741663+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:56.023911+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":156,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:37.233887+0000","last_change":"2024-03-30T13:19:23.950933+0000","last_active":"2024-03-30T13:19:37.233887+0000","last_peered":"2024-03-30T13:19:37.233887+0000","last_clean":"2024-03-30T13:19:37.233887+0000","last_became_active":"2024-03-30T13:19:23.949921+0000","last_became_peered":"2024-03-30T13:19:23.949921+0000","last_unstale":"2024-03-30T13:19:37.233887+0000","last_undegraded":"2024-03-30T13:19:37.233887+0000","last_fullsized":"2024-03-30T13:19:37.233887+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:18:58.851088+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:18:58.851088+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:07.550478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":354,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:37.233664+0000","last_change":"2024-03-30T13:19:16.856148+0000","last_active":"2024-03-30T13:19:37.233664+0000","last_peered":"2024-03-30T13:19:37.233664+0000","last_clean":"2024-03-30T13:19:37.233664+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:19:37.233664+0000","last_undegraded":"2024-03-30T13:19:37.233664+0000","last_fullsized":"2024-03-30T13:19:37.233664+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:19:16.855901+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:19:16.855901+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:19.845199+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1516,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:36.567998+0000","last_change":"2024-03-30T13:18:42.729125+0000","last_active":"2024-03-30T13:19:36.567998+0000","last_peered":"2024-03-30T13:19:36.567998+0000","last_clean":"2024-03-30T13:19:36.567998+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:19:36.567998+0000","last_undegraded":"2024-03-30T13:19:36.567998+0000","last_fullsized":"2024-03-30T13:19:36.567998+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_clean_scrub_stamp":"2024-03-30T13:18:42.729016+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:03.586884+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":191,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:37.234020+0000","last_change":"2024-03-30T13:19:35.780363+0000","last_active":"2024-03-30T13:19:37.234020+0000","last_peered":"2024-03-30T13:19:37.234020+0000","last_clean":"2024-03-30T13:19:37.234020+0000","last_became_active":"2024-03-30T13:19:23.388140+0000","last_became_peered":"2024-03-30T13:19:23.388140+0000","last_unstale":"2024-03-30T13:19:37.234020+0000","last_undegraded":"2024-03-30T13:19:37.234020+0000","last_fullsized":"2024-03-30T13:19:37.234020+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:19:35.780073+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:19:35.780073+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:59.577918+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":761,"reported_epoch":662,"state":"active+clean","last_fresh":"2024-03-30T13:19:34.495472+0000","last_change":"2024-03-30T13:18:59.894546+0000","last_active":"2024-03-30T13:19:34.495472+0000","last_peered":"2024-03-30T13:19:34.495472+0000","last_clean":"2024-03-30T13:19:34.495472+0000","last_became_active":"2024-03-30T13:18:32.413904+0000","last_became_peered":"2024-03-30T13:18:32.413904+0000","last_unstale":"2024-03-30T13:19:34.495472+0000","last_undegraded":"2024-03-30T13:19:34.495472+0000","last_fullsized":"2024-03-30T13:19:34.495472+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:18:59.894294+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:18:59.894294+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:21.905125+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1671,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:36.567961+0000","last_change":"2024-03-30T13:19:23.380909+0000","last_active":"2024-03-30T13:19:36.567961+0000","last_peered":"2024-03-30T13:19:36.567961+0000","last_clean":"2024-03-30T13:19:36.567961+0000","last_became_active":"2024-03-30T13:19:23.380536+0000","last_became_peered":"2024-03-30T13:19:23.380536+0000","last_unstale":"2024-03-30T13:19:36.567961+0000","last_undegraded":"2024-03-30T13:19:36.567961+0000","last_fullsized":"2024-03-30T13:19:36.567961+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:18:14.814123+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:18:14.814123+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":86286016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1270,"reported_epoch":661,"state":"active+clean","last_fresh":"2024-03-30T13:19:33.487230+0000","last_change":"2024-03-30T13:19:03.699231+0000","last_active":"2024-03-30T13:19:33.487230+0000","last_peered":"2024-03-30T13:19:33.487230+0000","last_clean":"2024-03-30T13:19:33.487230+0000","last_became_active":"2024-03-30T13:18:32.416688+0000","last_became_peered":"2024-03-30T13:18:32.416688+0000","last_unstale":"2024-03-30T13:19:33.487230+0000","last_undegraded":"2024-03-30T13:19:33.487230+0000","last_fullsized":"2024-03-30T13:19:33.487230+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:19:03.698837+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_clean_scrub_stamp":"2024-03-30T13:19:03.698837+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:21.113435+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1511,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T13:19:35.777871+0000","last_change":"2024-03-30T13:19:19.392352+0000","last_active":"2024-03-30T13:19:35.777871+0000","last_peered":"2024-03-30T13:19:35.777871+0000","last_clean":"2024-03-30T13:19:35.777871+0000","last_became_active":"2024-03-30T13:18:32.412897+0000","last_became_peered":"2024-03-30T13:18:32.412897+0000","last_unstale":"2024-03-30T13:19:35.777871+0000","last_undegraded":"2024-03-30T13:19:35.777871+0000","last_fullsized":"2024-03-30T13:19:35.777871+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:19:19.392102+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_clean_scrub_stamp":"2024-03-30T13:19:19.392102+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:22.733398+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1562,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T13:19:35.778090+0000","last_change":"2024-03-30T13:19:05.199227+0000","last_active":"2024-03-30T13:19:35.778090+0000","last_peered":"2024-03-30T13:19:35.778090+0000","last_clean":"2024-03-30T13:19:35.778090+0000","last_became_active":"2024-03-30T13:18:32.419902+0000","last_became_peered":"2024-03-30T13:18:32.419902+0000","last_unstale":"2024-03-30T13:19:35.778090+0000","last_undegraded":"2024-03-30T13:19:35.778090+0000","last_fullsized":"2024-03-30T13:19:35.778090+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:19:05.198795+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:19:05.198795+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:32.312402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":367,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:37.233650+0000","last_change":"2024-03-30T13:19:00.934425+0000","last_active":"2024-03-30T13:19:37.233650+0000","last_peered":"2024-03-30T13:19:37.233650+0000","last_clean":"2024-03-30T13:19:37.233650+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:19:37.233650+0000","last_undegraded":"2024-03-30T13:19:37.233650+0000","last_fullsized":"2024-03-30T13:19:37.233650+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:19:00.934164+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:19:00.934164+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:24.127933+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":147,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:37.234130+0000","last_change":"2024-03-30T13:19:23.382202+0000","last_active":"2024-03-30T13:19:37.234130+0000","last_peered":"2024-03-30T13:19:37.234130+0000","last_clean":"2024-03-30T13:19:37.234130+0000","last_became_active":"2024-03-30T13:19:23.380857+0000","last_became_peered":"2024-03-30T13:19:23.380857+0000","last_unstale":"2024-03-30T13:19:37.234130+0000","last_undegraded":"2024-03-30T13:19:37.234130+0000","last_fullsized":"2024-03-30T13:19:37.234130+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:18:35.867018+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:18:35.867018+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:45.320201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":373,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:37.233661+0000","last_change":"2024-03-30T13:19:33.837451+0000","last_active":"2024-03-30T13:19:37.233661+0000","last_peered":"2024-03-30T13:19:37.233661+0000","last_clean":"2024-03-30T13:19:37.233661+0000","last_became_active":"2024-03-30T13:18:32.415413+0000","last_became_peered":"2024-03-30T13:18:32.415413+0000","last_unstale":"2024-03-30T13:19:37.233661+0000","last_undegraded":"2024-03-30T13:19:37.233661+0000","last_fullsized":"2024-03-30T13:19:37.233661+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:19:33.837082+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:19:33.837082+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:54.376722+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":722,"reported_epoch":662,"state":"active+clean","last_fresh":"2024-03-30T13:19:34.495751+0000","last_change":"2024-03-30T13:19:19.587739+0000","last_active":"2024-03-30T13:19:34.495751+0000","last_peered":"2024-03-30T13:19:34.495751+0000","last_clean":"2024-03-30T13:19:34.495751+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:19:34.495751+0000","last_undegraded":"2024-03-30T13:19:34.495751+0000","last_fullsized":"2024-03-30T13:19:34.495751+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:19:19.587471+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:19:19.587471+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:40.749837+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":247967145,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":629,"seq":2701534429199,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59772,"kb_used_data":23016,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94312068,"statfs":{"total":96636764160,"available":96575557632,"internally_reserved":0,"allocated":23568384,"data_stored":19123968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157502,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65408,"kb_used_data":27760,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94306432,"statfs":{"total":96636764160,"available":96569786368,"internally_reserved":0,"allocated":28426240,"data_stored":23981824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":559,"seq":2400886718505,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57084,"kb_used_data":19440,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94314756,"statfs":{"total":96636764160,"available":96578310144,"internally_reserved":0,"allocated":19906560,"data_stored":15475746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12639,"internal_metadata":38522529},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509732,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49556,"kb_used_data":14208,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94322284,"statfs":{"total":96636764160,"available":96586018816,"internally_reserved":0,"allocated":14548992,"data_stored":10118178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375999011,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50692,"kb_used_data":13496,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94321148,"statfs":{"total":96636764160,"available":96584855552,"internally_reserved":0,"allocated":13819904,"data_stored":9382289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509733,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56732,"kb_used_data":21384,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315108,"statfs":{"total":96636764160,"available":96578670592,"internally_reserved":0,"allocated":21897216,"data_stored":17466402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742405,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54380,"kb_used_data":17952,"kb_used_omap":15,"kb_used_meta":36400,"kb_avail":94317460,"statfs":{"total":96636764160,"available":96581079040,"internally_reserved":0,"allocated":18382848,"data_stored":13938432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":37274051},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147442,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52116,"kb_used_data":20420,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94319724,"statfs":{"total":96636764160,"available":96583397376,"internally_reserved":0,"allocated":20910080,"data_stored":16458863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:19:39.206 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:19:39.206 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:19:39.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:39.307+0000 7f9fedea9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:39.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:39.323+0000 7f9fedea9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:39.496 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:19:39.496 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":666,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:19:38.541945+0000","last_up_change":"2024-03-30T13:19:22.353730+0000","last_in_change":"2024-03-30T13:19:22.353730+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":33,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":598,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":461,"last_clean_end":575,"up_from":581,"up_thru":646,"down_at":576,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6818","nonce":4030404750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6821","nonce":4030404750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6828","nonce":4030404750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6825","nonce":4030404750}]},"public_addr":"172.21.15.67:6818/4030404750","cluster_addr":"172.21.15.67:6821/4030404750","heartbeat_back_addr":"172.21.15.67:6828/4030404750","heartbeat_front_addr":"172.21.15.67:6825/4030404750","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":646,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":388,"up_from":393,"up_thru":650,"down_at":389,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6801","nonce":3837194718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6803","nonce":3837194718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6807","nonce":3837194718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6805","nonce":3837194718}]},"public_addr":"172.21.15.67:6801/3837194718","cluster_addr":"172.21.15.67:6803/3837194718","heartbeat_back_addr":"172.21.15.67:6807/3837194718","heartbeat_front_addr":"172.21.15.67:6805/3837194718","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":536,"last_clean_end":552,"up_from":559,"up_thru":650,"down_at":553,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":3095958959},{"type":"v1","addr":"172.21.15.82:6803","nonce":3095958959}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":3095958959},{"type":"v1","addr":"172.21.15.82:6807","nonce":3095958959}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":3095958959},{"type":"v1","addr":"172.21.15.82:6815","nonce":3095958959}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":3095958959},{"type":"v1","addr":"172.21.15.82:6811","nonce":3095958959}]},"public_addr":"172.21.15.82:6803/3095958959","cluster_addr":"172.21.15.82:6807/3095958959","heartbeat_back_addr":"172.21.15.82:6815/3095958959","heartbeat_front_addr":"172.21.15.82:6811/3095958959","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":488,"last_clean_end":506,"up_from":515,"up_thru":646,"down_at":507,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6817","nonce":292505453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6819","nonce":292505453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6823","nonce":292505453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6821","nonce":292505453}]},"public_addr":"172.21.15.82:6817/292505453","cluster_addr":"172.21.15.82:6819/292505453","heartbeat_back_addr":"172.21.15.82:6823/292505453","heartbeat_front_addr":"172.21.15.82:6821/292505453","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":372,"last_clean_end":621,"up_from":629,"up_thru":629,"down_at":622,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3888194455},{"type":"v1","addr":"172.21.15.82:6801","nonce":3888194455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3888194455},{"type":"v1","addr":"172.21.15.82:6806","nonce":3888194455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3888194455},{"type":"v1","addr":"172.21.15.82:6814","nonce":3888194455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3888194455},{"type":"v1","addr":"172.21.15.82:6810","nonce":3888194455}]},"public_addr":"172.21.15.82:6801/3888194455","cluster_addr":"172.21.15.82:6806/3888194455","heartbeat_back_addr":"172.21.15.82:6814/3888194455","heartbeat_front_addr":"172.21.15.82:6810/3888194455","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":604,"last_clean_end":645,"up_from":650,"up_thru":650,"down_at":646,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6825","nonce":1497758647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6827","nonce":1497758647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6831","nonce":1497758647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6829","nonce":1497758647}]},"public_addr":"172.21.15.82:6825/1497758647","cluster_addr":"172.21.15.82:6827/1497758647","heartbeat_back_addr":"172.21.15.82:6831/1497758647","heartbeat_front_addr":"172.21.15.82:6829/1497758647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:18:36.683094+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:18:42.615259+0000","dead_epoch":576},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:18:25.559437+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:18:37.671848+0000","dead_epoch":389},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:19:15.500985+0000","dead_epoch":553},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:18:41.029277+0000","dead_epoch":507},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:18:31.148522+0000","dead_epoch":622},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:19:21.682043+0000","dead_epoch":646}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:19:39.514 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:19:39.514 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:19:39.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:39.615+0000 7f755be40640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:39.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:39.631+0000 7f755be40640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:39.805 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:19:39.823 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:19:39.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:39.935+0000 7fb5eb0d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:39.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:39.951+0000 7fb5eb0d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:40.134 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:19:40.134 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:19:40.153 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:19:40.153 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:19:40.153 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:19:40.153 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:19:40.153 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 3, 5, 4, 1, 6, 7] 2024-03-30T13:19:40.154 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:19:40.154 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:19:40.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:40.263+0000 7fa1c1607640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:40.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:40.275+0000 7fa1c1607640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:40.445 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:19:40.446 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:19:40.462 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1383,"stamp":"2024-03-30T13:19:40.202456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":247967145,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":445928,"kb_used_data":157868,"kb_used_omap":62,"kb_used_meta":287873,"kb_avail":754528792,"statfs":{"total":773094113280,"available":772637483008,"internally_reserved":0,"allocated":161656832,"data_stored":126033494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64197,"internal_metadata":294782267},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.697162"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1534,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T13:19:35.778196+0000","last_change":"2024-03-30T13:18:57.255371+0000","last_active":"2024-03-30T13:19:35.778196+0000","last_peered":"2024-03-30T13:19:35.778196+0000","last_clean":"2024-03-30T13:19:35.778196+0000","last_became_active":"2024-03-30T13:18:32.416374+0000","last_became_peered":"2024-03-30T13:18:32.416374+0000","last_unstale":"2024-03-30T13:19:35.778196+0000","last_undegraded":"2024-03-30T13:19:35.778196+0000","last_fullsized":"2024-03-30T13:19:35.778196+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:18:57.255102+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:18:57.255102+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:01.922986+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1432,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-03-30T13:19:38.077451+0000","last_change":"2024-03-30T13:19:23.389925+0000","last_active":"2024-03-30T13:19:38.077451+0000","last_peered":"2024-03-30T13:19:38.077451+0000","last_clean":"2024-03-30T13:19:38.077451+0000","last_became_active":"2024-03-30T13:19:23.388425+0000","last_became_peered":"2024-03-30T13:19:23.388425+0000","last_unstale":"2024-03-30T13:19:38.077451+0000","last_undegraded":"2024-03-30T13:19:38.077451+0000","last_fullsized":"2024-03-30T13:19:38.077451+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:18:33.122892+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:18:33.122892+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:54.688629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10621892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":149,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:37.233852+0000","last_change":"2024-03-30T13:19:30.741906+0000","last_active":"2024-03-30T13:19:37.233852+0000","last_peered":"2024-03-30T13:19:37.233852+0000","last_clean":"2024-03-30T13:19:37.233852+0000","last_became_active":"2024-03-30T13:19:23.383364+0000","last_became_peered":"2024-03-30T13:19:23.383364+0000","last_unstale":"2024-03-30T13:19:37.233852+0000","last_undegraded":"2024-03-30T13:19:37.233852+0000","last_fullsized":"2024-03-30T13:19:37.233852+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:19:30.741663+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:19:30.741663+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:56.023911+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":156,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:37.233887+0000","last_change":"2024-03-30T13:19:23.950933+0000","last_active":"2024-03-30T13:19:37.233887+0000","last_peered":"2024-03-30T13:19:37.233887+0000","last_clean":"2024-03-30T13:19:37.233887+0000","last_became_active":"2024-03-30T13:19:23.949921+0000","last_became_peered":"2024-03-30T13:19:23.949921+0000","last_unstale":"2024-03-30T13:19:37.233887+0000","last_undegraded":"2024-03-30T13:19:37.233887+0000","last_fullsized":"2024-03-30T13:19:37.233887+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:18:58.851088+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:18:58.851088+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:07.550478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":354,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:37.233664+0000","last_change":"2024-03-30T13:19:16.856148+0000","last_active":"2024-03-30T13:19:37.233664+0000","last_peered":"2024-03-30T13:19:37.233664+0000","last_clean":"2024-03-30T13:19:37.233664+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:19:37.233664+0000","last_undegraded":"2024-03-30T13:19:37.233664+0000","last_fullsized":"2024-03-30T13:19:37.233664+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:19:16.855901+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:19:16.855901+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:19.845199+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1516,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:36.567998+0000","last_change":"2024-03-30T13:18:42.729125+0000","last_active":"2024-03-30T13:19:36.567998+0000","last_peered":"2024-03-30T13:19:36.567998+0000","last_clean":"2024-03-30T13:19:36.567998+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:19:36.567998+0000","last_undegraded":"2024-03-30T13:19:36.567998+0000","last_fullsized":"2024-03-30T13:19:36.567998+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_clean_scrub_stamp":"2024-03-30T13:18:42.729016+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:03.586884+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":191,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:37.234020+0000","last_change":"2024-03-30T13:19:35.780363+0000","last_active":"2024-03-30T13:19:37.234020+0000","last_peered":"2024-03-30T13:19:37.234020+0000","last_clean":"2024-03-30T13:19:37.234020+0000","last_became_active":"2024-03-30T13:19:23.388140+0000","last_became_peered":"2024-03-30T13:19:23.388140+0000","last_unstale":"2024-03-30T13:19:37.234020+0000","last_undegraded":"2024-03-30T13:19:37.234020+0000","last_fullsized":"2024-03-30T13:19:37.234020+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:19:35.780073+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:19:35.780073+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:59.577918+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":769,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-03-30T13:19:39.508065+0000","last_change":"2024-03-30T13:18:59.894546+0000","last_active":"2024-03-30T13:19:39.508065+0000","last_peered":"2024-03-30T13:19:39.508065+0000","last_clean":"2024-03-30T13:19:39.508065+0000","last_became_active":"2024-03-30T13:18:32.413904+0000","last_became_peered":"2024-03-30T13:18:32.413904+0000","last_unstale":"2024-03-30T13:19:39.508065+0000","last_undegraded":"2024-03-30T13:19:39.508065+0000","last_fullsized":"2024-03-30T13:19:39.508065+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:18:59.894294+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:18:59.894294+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:21.905125+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1671,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:36.567961+0000","last_change":"2024-03-30T13:19:23.380909+0000","last_active":"2024-03-30T13:19:36.567961+0000","last_peered":"2024-03-30T13:19:36.567961+0000","last_clean":"2024-03-30T13:19:36.567961+0000","last_became_active":"2024-03-30T13:19:23.380536+0000","last_became_peered":"2024-03-30T13:19:23.380536+0000","last_unstale":"2024-03-30T13:19:36.567961+0000","last_undegraded":"2024-03-30T13:19:36.567961+0000","last_fullsized":"2024-03-30T13:19:36.567961+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:18:14.814123+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:18:14.814123+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":86286016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1278,"reported_epoch":665,"state":"active+clean","last_fresh":"2024-03-30T13:19:38.077328+0000","last_change":"2024-03-30T13:19:03.699231+0000","last_active":"2024-03-30T13:19:38.077328+0000","last_peered":"2024-03-30T13:19:38.077328+0000","last_clean":"2024-03-30T13:19:38.077328+0000","last_became_active":"2024-03-30T13:18:32.416688+0000","last_became_peered":"2024-03-30T13:18:32.416688+0000","last_unstale":"2024-03-30T13:19:38.077328+0000","last_undegraded":"2024-03-30T13:19:38.077328+0000","last_fullsized":"2024-03-30T13:19:38.077328+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:19:03.698837+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_clean_scrub_stamp":"2024-03-30T13:19:03.698837+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:21.113435+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1511,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T13:19:35.777871+0000","last_change":"2024-03-30T13:19:19.392352+0000","last_active":"2024-03-30T13:19:35.777871+0000","last_peered":"2024-03-30T13:19:35.777871+0000","last_clean":"2024-03-30T13:19:35.777871+0000","last_became_active":"2024-03-30T13:18:32.412897+0000","last_became_peered":"2024-03-30T13:18:32.412897+0000","last_unstale":"2024-03-30T13:19:35.777871+0000","last_undegraded":"2024-03-30T13:19:35.777871+0000","last_fullsized":"2024-03-30T13:19:35.777871+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:19:19.392102+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_clean_scrub_stamp":"2024-03-30T13:19:19.392102+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:22.733398+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1562,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-03-30T13:19:35.778090+0000","last_change":"2024-03-30T13:19:05.199227+0000","last_active":"2024-03-30T13:19:35.778090+0000","last_peered":"2024-03-30T13:19:35.778090+0000","last_clean":"2024-03-30T13:19:35.778090+0000","last_became_active":"2024-03-30T13:18:32.419902+0000","last_became_peered":"2024-03-30T13:18:32.419902+0000","last_unstale":"2024-03-30T13:19:35.778090+0000","last_undegraded":"2024-03-30T13:19:35.778090+0000","last_fullsized":"2024-03-30T13:19:35.778090+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:19:05.198795+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:19:05.198795+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:32.312402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":367,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:37.233650+0000","last_change":"2024-03-30T13:19:00.934425+0000","last_active":"2024-03-30T13:19:37.233650+0000","last_peered":"2024-03-30T13:19:37.233650+0000","last_clean":"2024-03-30T13:19:37.233650+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:19:37.233650+0000","last_undegraded":"2024-03-30T13:19:37.233650+0000","last_fullsized":"2024-03-30T13:19:37.233650+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:19:00.934164+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:19:00.934164+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:24.127933+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":147,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:37.234130+0000","last_change":"2024-03-30T13:19:23.382202+0000","last_active":"2024-03-30T13:19:37.234130+0000","last_peered":"2024-03-30T13:19:37.234130+0000","last_clean":"2024-03-30T13:19:37.234130+0000","last_became_active":"2024-03-30T13:19:23.380857+0000","last_became_peered":"2024-03-30T13:19:23.380857+0000","last_unstale":"2024-03-30T13:19:37.234130+0000","last_undegraded":"2024-03-30T13:19:37.234130+0000","last_fullsized":"2024-03-30T13:19:37.234130+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:18:35.867018+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:18:35.867018+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:45.320201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":373,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:37.233661+0000","last_change":"2024-03-30T13:19:33.837451+0000","last_active":"2024-03-30T13:19:37.233661+0000","last_peered":"2024-03-30T13:19:37.233661+0000","last_clean":"2024-03-30T13:19:37.233661+0000","last_became_active":"2024-03-30T13:18:32.415413+0000","last_became_peered":"2024-03-30T13:18:32.415413+0000","last_unstale":"2024-03-30T13:19:37.233661+0000","last_undegraded":"2024-03-30T13:19:37.233661+0000","last_fullsized":"2024-03-30T13:19:37.233661+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:19:33.837082+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:19:33.837082+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:54.376722+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":730,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-03-30T13:19:39.508071+0000","last_change":"2024-03-30T13:19:19.587739+0000","last_active":"2024-03-30T13:19:39.508071+0000","last_peered":"2024-03-30T13:19:39.508071+0000","last_clean":"2024-03-30T13:19:39.508071+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:19:39.508071+0000","last_undegraded":"2024-03-30T13:19:39.508071+0000","last_fullsized":"2024-03-30T13:19:39.508071+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:19:19.587471+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:19:19.587471+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:40.749837+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":247967145,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":629,"seq":2701534429199,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59772,"kb_used_data":23016,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94312068,"statfs":{"total":96636764160,"available":96575557632,"internally_reserved":0,"allocated":23568384,"data_stored":19123968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157502,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65408,"kb_used_data":27760,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94306432,"statfs":{"total":96636764160,"available":96569786368,"internally_reserved":0,"allocated":28426240,"data_stored":23981824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":559,"seq":2400886718506,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57136,"kb_used_data":19488,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94314704,"statfs":{"total":96636764160,"available":96578256896,"internally_reserved":0,"allocated":19955712,"data_stored":15497694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12639,"internal_metadata":38522529},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509733,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49612,"kb_used_data":14268,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94322228,"statfs":{"total":96636764160,"available":96585961472,"internally_reserved":0,"allocated":14610432,"data_stored":10145613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375999012,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50732,"kb_used_data":13532,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94321108,"statfs":{"total":96636764160,"available":96584814592,"internally_reserved":0,"allocated":13856768,"data_stored":9398750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509734,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56772,"kb_used_data":21432,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315068,"statfs":{"total":96636764160,"available":96578629632,"internally_reserved":0,"allocated":21946368,"data_stored":17488350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742405,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54380,"kb_used_data":17952,"kb_used_omap":15,"kb_used_meta":36400,"kb_avail":94317460,"statfs":{"total":96636764160,"available":96581079040,"internally_reserved":0,"allocated":18382848,"data_stored":13938432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":37274051},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147442,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52116,"kb_used_data":20420,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94319724,"statfs":{"total":96636764160,"available":96583397376,"internally_reserved":0,"allocated":20910080,"data_stored":16458863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:19:40.463 INFO:tasks.thrashosds.thrasher:Choosing PG 1.5 with acting set [4, 2, 6] 2024-03-30T13:19:40.463 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [0, 2, 3, 5, 4, 1, 6, 7] 2024-03-30T13:19:40.464 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-30T13:19:40.464 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:19:40.525 INFO:tasks.ceph.osd.4:Stopped 2024-03-30T13:19:40.525 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 2, 3, 5, 4, 1, 6, 7] 2024-03-30T13:19:40.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 4 2024-03-30T13:19:40.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:40.635+0000 7fc38be9a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:40.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:40.647+0000 7fc38be9a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:40.826 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:19:40.823+0000 7fbb38f08640 -1 mon.a@0(leader).osd e667 definitely_dead 0 2024-03-30T13:19:41.561 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.4. 2024-03-30T13:19:41.590 INFO:tasks.thrashosds.thrasher:dead_osds=[4], live_osds=[0, 2, 3, 5, 1, 6, 7] 2024-03-30T13:19:41.590 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:19:41.591 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:19:41.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:41.699+0000 7f73616eb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:41.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:19:41.711+0000 7f73616eb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:19:41.886 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:19:41.887 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:19:41.905 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1385,"stamp":"2024-03-30T13:19:41.566314+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":247967145,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":388836,"kb_used_data":138428,"kb_used_omap":50,"kb_used_meta":250253,"kb_avail":660214044,"statfs":{"total":676457349120,"available":676059181056,"internally_reserved":0,"allocated":141750272,"data_stored":110557924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":51558,"internal_metadata":256259738},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.052016"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1541,"reported_epoch":666,"state":"peering","last_fresh":"2024-03-30T13:19:39.778271+0000","last_change":"2024-03-30T13:19:40.570474+0000","last_active":"2024-03-30T13:19:40.570474+0000","last_peered":"2024-03-30T13:19:39.778271+0000","last_clean":"2024-03-30T13:19:39.778271+0000","last_became_active":"2024-03-30T13:18:32.416374+0000","last_became_peered":"2024-03-30T13:18:32.416374+0000","last_unstale":"2024-03-30T13:19:39.778271+0000","last_undegraded":"2024-03-30T13:19:39.778271+0000","last_fullsized":"2024-03-30T13:19:39.778271+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:18:57.255102+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:18:57.255102+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2147483647],"acting":[6,5,2147483647],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1432,"reported_epoch":665,"state":"stale+active+clean","last_fresh":"2024-03-30T13:19:38.077451+0000","last_change":"2024-03-30T13:19:23.389925+0000","last_active":"2024-03-30T13:19:38.077451+0000","last_peered":"2024-03-30T13:19:38.077451+0000","last_clean":"2024-03-30T13:19:38.077451+0000","last_became_active":"2024-03-30T13:19:23.388425+0000","last_became_peered":"2024-03-30T13:19:23.388425+0000","last_unstale":"2024-03-30T13:19:40.563711+0000","last_undegraded":"2024-03-30T13:19:38.077451+0000","last_fullsized":"2024-03-30T13:19:38.077451+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:18:33.122892+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:18:33.122892+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:54.688629+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":10621892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":149,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:37.233852+0000","last_change":"2024-03-30T13:19:30.741906+0000","last_active":"2024-03-30T13:19:37.233852+0000","last_peered":"2024-03-30T13:19:37.233852+0000","last_clean":"2024-03-30T13:19:37.233852+0000","last_became_active":"2024-03-30T13:19:23.383364+0000","last_became_peered":"2024-03-30T13:19:23.383364+0000","last_unstale":"2024-03-30T13:19:37.233852+0000","last_undegraded":"2024-03-30T13:19:37.233852+0000","last_fullsized":"2024-03-30T13:19:37.233852+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:19:30.741663+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:19:30.741663+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:56.023911+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":15630312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":156,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:37.233887+0000","last_change":"2024-03-30T13:19:23.950933+0000","last_active":"2024-03-30T13:19:37.233887+0000","last_peered":"2024-03-30T13:19:37.233887+0000","last_clean":"2024-03-30T13:19:37.233887+0000","last_became_active":"2024-03-30T13:19:23.949921+0000","last_became_peered":"2024-03-30T13:19:23.949921+0000","last_unstale":"2024-03-30T13:19:37.233887+0000","last_undegraded":"2024-03-30T13:19:37.233887+0000","last_fullsized":"2024-03-30T13:19:37.233887+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:18:58.851088+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:18:58.851088+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:07.550478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":354,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:37.233664+0000","last_change":"2024-03-30T13:19:16.856148+0000","last_active":"2024-03-30T13:19:37.233664+0000","last_peered":"2024-03-30T13:19:37.233664+0000","last_clean":"2024-03-30T13:19:37.233664+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:19:37.233664+0000","last_undegraded":"2024-03-30T13:19:37.233664+0000","last_fullsized":"2024-03-30T13:19:37.233664+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:19:16.855901+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:19:16.855901+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:19.845199+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1516,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:36.567998+0000","last_change":"2024-03-30T13:18:42.729125+0000","last_active":"2024-03-30T13:19:36.567998+0000","last_peered":"2024-03-30T13:19:36.567998+0000","last_clean":"2024-03-30T13:19:36.567998+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:19:36.567998+0000","last_undegraded":"2024-03-30T13:19:36.567998+0000","last_fullsized":"2024-03-30T13:19:36.567998+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_clean_scrub_stamp":"2024-03-30T13:18:42.729016+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:03.586884+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":191,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:37.234020+0000","last_change":"2024-03-30T13:19:35.780363+0000","last_active":"2024-03-30T13:19:37.234020+0000","last_peered":"2024-03-30T13:19:37.234020+0000","last_clean":"2024-03-30T13:19:37.234020+0000","last_became_active":"2024-03-30T13:19:23.388140+0000","last_became_peered":"2024-03-30T13:19:23.388140+0000","last_unstale":"2024-03-30T13:19:37.234020+0000","last_undegraded":"2024-03-30T13:19:37.234020+0000","last_fullsized":"2024-03-30T13:19:37.234020+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:19:35.780073+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:19:35.780073+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:59.577918+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":769,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-03-30T13:19:39.508065+0000","last_change":"2024-03-30T13:18:59.894546+0000","last_active":"2024-03-30T13:19:39.508065+0000","last_peered":"2024-03-30T13:19:39.508065+0000","last_clean":"2024-03-30T13:19:39.508065+0000","last_became_active":"2024-03-30T13:18:32.413904+0000","last_became_peered":"2024-03-30T13:18:32.413904+0000","last_unstale":"2024-03-30T13:19:39.508065+0000","last_undegraded":"2024-03-30T13:19:39.508065+0000","last_fullsized":"2024-03-30T13:19:39.508065+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:18:59.894294+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:18:59.894294+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:21.905125+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1671,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:36.567961+0000","last_change":"2024-03-30T13:19:23.380909+0000","last_active":"2024-03-30T13:19:36.567961+0000","last_peered":"2024-03-30T13:19:36.567961+0000","last_clean":"2024-03-30T13:19:36.567961+0000","last_became_active":"2024-03-30T13:19:23.380536+0000","last_became_peered":"2024-03-30T13:19:23.380536+0000","last_unstale":"2024-03-30T13:19:36.567961+0000","last_undegraded":"2024-03-30T13:19:36.567961+0000","last_fullsized":"2024-03-30T13:19:36.567961+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:18:14.814123+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:18:14.814123+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":86286016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1278,"reported_epoch":665,"state":"stale+active+clean","last_fresh":"2024-03-30T13:19:38.077328+0000","last_change":"2024-03-30T13:19:03.699231+0000","last_active":"2024-03-30T13:19:38.077328+0000","last_peered":"2024-03-30T13:19:38.077328+0000","last_clean":"2024-03-30T13:19:38.077328+0000","last_became_active":"2024-03-30T13:18:32.416688+0000","last_became_peered":"2024-03-30T13:18:32.416688+0000","last_unstale":"2024-03-30T13:19:40.563717+0000","last_undegraded":"2024-03-30T13:19:38.077328+0000","last_fullsized":"2024-03-30T13:19:38.077328+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:19:03.698837+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_clean_scrub_stamp":"2024-03-30T13:19:03.698837+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:21.113435+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1518,"reported_epoch":666,"state":"peering","last_fresh":"2024-03-30T13:19:39.778011+0000","last_change":"2024-03-30T13:19:40.569280+0000","last_active":"2024-03-30T13:19:40.569280+0000","last_peered":"2024-03-30T13:19:39.778011+0000","last_clean":"2024-03-30T13:19:39.778011+0000","last_became_active":"2024-03-30T13:18:32.412897+0000","last_became_peered":"2024-03-30T13:18:32.412897+0000","last_unstale":"2024-03-30T13:19:39.778011+0000","last_undegraded":"2024-03-30T13:19:39.778011+0000","last_fullsized":"2024-03-30T13:19:39.778011+0000","mapping_epoch":667,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:19:19.392102+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_clean_scrub_stamp":"2024-03-30T13:19:19.392102+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2147483647],"acting":[6,0,2147483647],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1570,"reported_epoch":667,"state":"active+clean","last_fresh":"2024-03-30T13:19:40.568917+0000","last_change":"2024-03-30T13:19:05.199227+0000","last_active":"2024-03-30T13:19:40.568917+0000","last_peered":"2024-03-30T13:19:40.568917+0000","last_clean":"2024-03-30T13:19:40.568917+0000","last_became_active":"2024-03-30T13:18:32.419902+0000","last_became_peered":"2024-03-30T13:18:32.419902+0000","last_unstale":"2024-03-30T13:19:40.568917+0000","last_undegraded":"2024-03-30T13:19:40.568917+0000","last_fullsized":"2024-03-30T13:19:40.568917+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:19:05.198795+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:19:05.198795+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:32.312402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":367,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:37.233650+0000","last_change":"2024-03-30T13:19:00.934425+0000","last_active":"2024-03-30T13:19:37.233650+0000","last_peered":"2024-03-30T13:19:37.233650+0000","last_clean":"2024-03-30T13:19:37.233650+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:19:37.233650+0000","last_undegraded":"2024-03-30T13:19:37.233650+0000","last_fullsized":"2024-03-30T13:19:37.233650+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:19:00.934164+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:19:00.934164+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:24.127933+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":147,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:37.234130+0000","last_change":"2024-03-30T13:19:23.382202+0000","last_active":"2024-03-30T13:19:37.234130+0000","last_peered":"2024-03-30T13:19:37.234130+0000","last_clean":"2024-03-30T13:19:37.234130+0000","last_became_active":"2024-03-30T13:19:23.380857+0000","last_became_peered":"2024-03-30T13:19:23.380857+0000","last_unstale":"2024-03-30T13:19:37.234130+0000","last_undegraded":"2024-03-30T13:19:37.234130+0000","last_fullsized":"2024-03-30T13:19:37.234130+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:18:35.867018+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:18:35.867018+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:19:45.320201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":373,"reported_epoch":664,"state":"active+clean","last_fresh":"2024-03-30T13:19:37.233661+0000","last_change":"2024-03-30T13:19:33.837451+0000","last_active":"2024-03-30T13:19:37.233661+0000","last_peered":"2024-03-30T13:19:37.233661+0000","last_clean":"2024-03-30T13:19:37.233661+0000","last_became_active":"2024-03-30T13:18:32.415413+0000","last_became_peered":"2024-03-30T13:18:32.415413+0000","last_unstale":"2024-03-30T13:19:37.233661+0000","last_undegraded":"2024-03-30T13:19:37.233661+0000","last_fullsized":"2024-03-30T13:19:37.233661+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:19:33.837082+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:19:33.837082+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:54.376722+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":730,"reported_epoch":666,"state":"active+clean","last_fresh":"2024-03-30T13:19:39.508071+0000","last_change":"2024-03-30T13:19:19.587739+0000","last_active":"2024-03-30T13:19:39.508071+0000","last_peered":"2024-03-30T13:19:39.508071+0000","last_clean":"2024-03-30T13:19:39.508071+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:19:39.508071+0000","last_undegraded":"2024-03-30T13:19:39.508071+0000","last_fullsized":"2024-03-30T13:19:39.508071+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:19:19.587471+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:19:19.587471+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:40.749837+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":247967145,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":629,"seq":2701534429200,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59816,"kb_used_data":23064,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94312024,"statfs":{"total":96636764160,"available":96575512576,"internally_reserved":0,"allocated":23617536,"data_stored":19146092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157502,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65408,"kb_used_data":27760,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94306432,"statfs":{"total":96636764160,"available":96569786368,"internally_reserved":0,"allocated":28426240,"data_stored":23981824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509733,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49612,"kb_used_data":14268,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94322228,"statfs":{"total":96636764160,"available":96585961472,"internally_reserved":0,"allocated":14610432,"data_stored":10145613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375999012,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50732,"kb_used_data":13532,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94321108,"statfs":{"total":96636764160,"available":96584814592,"internally_reserved":0,"allocated":13856768,"data_stored":9398750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509734,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56772,"kb_used_data":21432,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315068,"statfs":{"total":96636764160,"available":96578629632,"internally_reserved":0,"allocated":21946368,"data_stored":17488350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742405,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54380,"kb_used_data":17952,"kb_used_omap":15,"kb_used_meta":36400,"kb_avail":94317460,"statfs":{"total":96636764160,"available":96581079040,"internally_reserved":0,"allocated":18382848,"data_stored":13938432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":37274051},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147442,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52116,"kb_used_data":20420,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94319724,"statfs":{"total":96636764160,"available":96583397376,"internally_reserved":0,"allocated":20910080,"data_stored":16458863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:19:41.906 INFO:tasks.ceph.ceph_manager.ceph:PG 1.f is not active or peered 2024-03-30T13:19:41.906 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.f', 'version': "22'6", 'reported_seq': 1541, 'reported_epoch': 666, 'state': 'peering', 'last_fresh': '2024-03-30T13:19:39.778271+0000', 'last_change': '2024-03-30T13:19:40.570474+0000', 'last_active': '2024-03-30T13:19:40.570474+0000', 'last_peered': '2024-03-30T13:19:39.778271+0000', 'last_clean': '2024-03-30T13:19:39.778271+0000', 'last_became_active': '2024-03-30T13:18:32.416374+0000', 'last_became_peered': '2024-03-30T13:18:32.416374+0000', 'last_unstale': '2024-03-30T13:19:39.778271+0000', 'last_undegraded': '2024-03-30T13:19:39.778271+0000', 'last_fullsized': '2024-03-30T13:19:39.778271+0000', 'mapping_epoch': 667, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 630, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "22'6", 'last_scrub_stamp': '2024-03-30T13:18:57.255102+0000', 'last_deep_scrub': "22'6", 'last_deep_scrub_stamp': '2024-03-30T13:17:53.569165+0000', 'last_clean_scrub_stamp': '2024-03-30T13:18:57.255102+0000', 'objects_scrubbed': 2, 'log_size': 6, 'log_dups_size': 0, 'ondisk_log_size': 6, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4588245, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 3, 'num_read_kb': 1, 'num_write': 8, 'num_write_kb': 1781, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2, 'num_bytes_recovered': 4588245, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5, 2147483647], 'acting': [6, 5, 2147483647], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [5], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-30T13:19:41.906 INFO:tasks.ceph.ceph_manager.ceph:PG 1.6 is not active or peered 2024-03-30T13:19:41.906 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.6', 'version': "22'15", 'reported_seq': 1518, 'reported_epoch': 666, 'state': 'peering', 'last_fresh': '2024-03-30T13:19:39.778011+0000', 'last_change': '2024-03-30T13:19:40.569280+0000', 'last_active': '2024-03-30T13:19:40.569280+0000', 'last_peered': '2024-03-30T13:19:39.778011+0000', 'last_clean': '2024-03-30T13:19:39.778011+0000', 'last_became_active': '2024-03-30T13:18:32.412897+0000', 'last_became_peered': '2024-03-30T13:18:32.412897+0000', 'last_unstale': '2024-03-30T13:19:39.778011+0000', 'last_undegraded': '2024-03-30T13:19:39.778011+0000', 'last_fullsized': '2024-03-30T13:19:39.778011+0000', 'mapping_epoch': 667, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 630, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "22'15", 'last_scrub_stamp': '2024-03-30T13:19:19.392102+0000', 'last_deep_scrub': "22'15", 'last_deep_scrub_stamp': '2024-03-30T13:17:50.531859+0000', 'last_clean_scrub_stamp': '2024-03-30T13:19:19.392102+0000', 'objects_scrubbed': 5, 'log_size': 15, 'log_dups_size': 0, 'ondisk_log_size': 15, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 7, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 12739541, 'num_objects': 5, 'num_object_clones': 0, 'num_object_copies': 15, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 5, 'num_whiteouts': 0, 'num_read': 9, 'num_read_kb': 4, 'num_write': 20, 'num_write_kb': 6289, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0, 2147483647], 'acting': [6, 0, 2147483647], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-30T13:19:41.907 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:20:06.907 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:20:06.908 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:20:07.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:07.030+0000 7f2c0cc30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:07.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:07.042+0000 7f2c0cc30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:07.221 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:20:07.222 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:20:07.239 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1404,"stamp":"2024-03-30T13:20:06.209626+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":277446227,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":401708,"kb_used_data":151324,"kb_used_omap":50,"kb_used_meta":250253,"kb_avail":660201172,"statfs":{"total":676457349120,"available":676046000128,"internally_reserved":0,"allocated":154955776,"data_stored":123378647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":51558,"internal_metadata":256259738},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002080"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1565,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-03-30T13:20:00.492845+0000","last_change":"2024-03-30T13:19:43.165673+0000","last_active":"2024-03-30T13:20:00.492845+0000","last_peered":"2024-03-30T13:20:00.492845+0000","last_clean":"2024-03-30T13:20:00.492845+0000","last_became_active":"2024-03-30T13:19:42.586512+0000","last_became_peered":"2024-03-30T13:19:42.586512+0000","last_unstale":"2024-03-30T13:20:00.492845+0000","last_undegraded":"2024-03-30T13:20:00.492845+0000","last_fullsized":"2024-03-30T13:20:00.492845+0000","mapping_epoch":668,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":669,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:18:57.255102+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:18:57.255102+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":31,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-03-30T13:20:00.339618+0000","last_change":"2024-03-30T13:19:43.319298+0000","last_active":"2024-03-30T13:20:00.339618+0000","last_peered":"2024-03-30T13:20:00.339618+0000","last_clean":"2024-03-30T13:20:00.339618+0000","last_became_active":"2024-03-30T13:19:42.581188+0000","last_became_peered":"2024-03-30T13:19:42.581188+0000","last_unstale":"2024-03-30T13:20:00.339618+0000","last_undegraded":"2024-03-30T13:20:00.339618+0000","last_fullsized":"2024-03-30T13:20:00.339618+0000","mapping_epoch":668,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":669,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:19:43.319043+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:19:43.319043+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:55.346489+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":180,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-03-30T13:20:00.499104+0000","last_change":"2024-03-30T13:19:42.679012+0000","last_active":"2024-03-30T13:20:00.499104+0000","last_peered":"2024-03-30T13:20:00.499104+0000","last_clean":"2024-03-30T13:20:00.499104+0000","last_became_active":"2024-03-30T13:19:42.583005+0000","last_became_peered":"2024-03-30T13:19:42.583005+0000","last_unstale":"2024-03-30T13:20:00.499104+0000","last_undegraded":"2024-03-30T13:20:00.499104+0000","last_fullsized":"2024-03-30T13:20:00.499104+0000","mapping_epoch":668,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":669,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:19:30.741663+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:19:30.741663+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:36.489122+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":174,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-03-30T13:20:00.499064+0000","last_change":"2024-03-30T13:19:23.950933+0000","last_active":"2024-03-30T13:20:00.499064+0000","last_peered":"2024-03-30T13:20:00.499064+0000","last_clean":"2024-03-30T13:20:00.499064+0000","last_became_active":"2024-03-30T13:19:23.949921+0000","last_became_peered":"2024-03-30T13:19:23.949921+0000","last_unstale":"2024-03-30T13:20:00.499064+0000","last_undegraded":"2024-03-30T13:20:00.499064+0000","last_fullsized":"2024-03-30T13:20:00.499064+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:18:58.851088+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:18:58.851088+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:07.550478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":372,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-03-30T13:20:00.338504+0000","last_change":"2024-03-30T13:19:16.856148+0000","last_active":"2024-03-30T13:20:00.338504+0000","last_peered":"2024-03-30T13:20:00.338504+0000","last_clean":"2024-03-30T13:20:00.338504+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:20:00.338504+0000","last_undegraded":"2024-03-30T13:20:00.338504+0000","last_fullsized":"2024-03-30T13:20:00.338504+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:19:16.855901+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:09:56.639030+0000","last_clean_scrub_stamp":"2024-03-30T13:19:16.855901+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:19.845199+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1534,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-03-30T13:20:00.498477+0000","last_change":"2024-03-30T13:18:42.729125+0000","last_active":"2024-03-30T13:20:00.498477+0000","last_peered":"2024-03-30T13:20:00.498477+0000","last_clean":"2024-03-30T13:20:00.498477+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:20:00.498477+0000","last_undegraded":"2024-03-30T13:20:00.498477+0000","last_fullsized":"2024-03-30T13:20:00.498477+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_clean_scrub_stamp":"2024-03-30T13:18:42.729016+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:03.586884+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":228,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-03-30T13:20:00.499096+0000","last_change":"2024-03-30T13:19:42.712501+0000","last_active":"2024-03-30T13:20:00.499096+0000","last_peered":"2024-03-30T13:20:00.499096+0000","last_clean":"2024-03-30T13:20:00.499096+0000","last_became_active":"2024-03-30T13:19:42.583777+0000","last_became_peered":"2024-03-30T13:19:42.583777+0000","last_unstale":"2024-03-30T13:20:00.499096+0000","last_undegraded":"2024-03-30T13:20:00.499096+0000","last_fullsized":"2024-03-30T13:20:00.499096+0000","mapping_epoch":668,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":669,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:19:35.780073+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:19:35.780073+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:35.784075+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":783,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-03-30T13:20:00.338861+0000","last_change":"2024-03-30T13:18:59.894546+0000","last_active":"2024-03-30T13:20:00.338861+0000","last_peered":"2024-03-30T13:20:00.338861+0000","last_clean":"2024-03-30T13:20:00.338861+0000","last_became_active":"2024-03-30T13:18:32.413904+0000","last_became_peered":"2024-03-30T13:18:32.413904+0000","last_unstale":"2024-03-30T13:20:00.338861+0000","last_undegraded":"2024-03-30T13:20:00.338861+0000","last_fullsized":"2024-03-30T13:20:00.338861+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:18:59.894294+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:18:59.894294+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:21.905125+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1697,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-03-30T13:20:00.498406+0000","last_change":"2024-03-30T13:19:37.564272+0000","last_active":"2024-03-30T13:20:00.498406+0000","last_peered":"2024-03-30T13:20:00.498406+0000","last_clean":"2024-03-30T13:20:00.498406+0000","last_became_active":"2024-03-30T13:19:23.380536+0000","last_became_peered":"2024-03-30T13:19:23.380536+0000","last_unstale":"2024-03-30T13:20:00.498406+0000","last_undegraded":"2024-03-30T13:20:00.498406+0000","last_fullsized":"2024-03-30T13:20:00.498406+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:19:37.564154+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:19:37.564154+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:51.343381+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":86286016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":23,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-03-30T13:20:00.497564+0000","last_change":"2024-03-30T13:19:42.612370+0000","last_active":"2024-03-30T13:20:00.497564+0000","last_peered":"2024-03-30T13:20:00.497564+0000","last_clean":"2024-03-30T13:20:00.497564+0000","last_became_active":"2024-03-30T13:19:42.578422+0000","last_became_peered":"2024-03-30T13:19:42.578422+0000","last_unstale":"2024-03-30T13:20:00.497564+0000","last_undegraded":"2024-03-30T13:20:00.497564+0000","last_fullsized":"2024-03-30T13:20:00.497564+0000","mapping_epoch":668,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":669,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:19:03.698837+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:16:16.623820+0000","last_clean_scrub_stamp":"2024-03-30T13:19:03.698837+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:28.806897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1548,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-03-30T13:20:00.492652+0000","last_change":"2024-03-30T13:19:42.707273+0000","last_active":"2024-03-30T13:20:00.492652+0000","last_peered":"2024-03-30T13:20:00.492652+0000","last_clean":"2024-03-30T13:20:00.492652+0000","last_became_active":"2024-03-30T13:19:42.584322+0000","last_became_peered":"2024-03-30T13:19:42.584322+0000","last_unstale":"2024-03-30T13:20:00.492652+0000","last_undegraded":"2024-03-30T13:20:00.492652+0000","last_fullsized":"2024-03-30T13:20:00.492652+0000","mapping_epoch":668,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":669,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:19:19.392102+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_clean_scrub_stamp":"2024-03-30T13:19:19.392102+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:23.827227+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1582,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-03-30T13:20:00.492466+0000","last_change":"2024-03-30T13:19:05.199227+0000","last_active":"2024-03-30T13:20:00.492466+0000","last_peered":"2024-03-30T13:20:00.492466+0000","last_clean":"2024-03-30T13:20:00.492466+0000","last_became_active":"2024-03-30T13:18:32.419902+0000","last_became_peered":"2024-03-30T13:18:32.419902+0000","last_unstale":"2024-03-30T13:20:00.492466+0000","last_undegraded":"2024-03-30T13:20:00.492466+0000","last_fullsized":"2024-03-30T13:20:00.492466+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:19:05.198795+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:19:05.198795+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:32.312402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":385,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-03-30T13:20:00.338495+0000","last_change":"2024-03-30T13:19:00.934425+0000","last_active":"2024-03-30T13:20:00.338495+0000","last_peered":"2024-03-30T13:20:00.338495+0000","last_clean":"2024-03-30T13:20:00.338495+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:20:00.338495+0000","last_undegraded":"2024-03-30T13:20:00.338495+0000","last_fullsized":"2024-03-30T13:20:00.338495+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:19:00.934164+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:19:00.934164+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:24.127933+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":173,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-03-30T13:20:00.499067+0000","last_change":"2024-03-30T13:19:48.759695+0000","last_active":"2024-03-30T13:20:00.499067+0000","last_peered":"2024-03-30T13:20:00.499067+0000","last_clean":"2024-03-30T13:20:00.499067+0000","last_became_active":"2024-03-30T13:19:23.380857+0000","last_became_peered":"2024-03-30T13:19:23.380857+0000","last_unstale":"2024-03-30T13:20:00.499067+0000","last_undegraded":"2024-03-30T13:20:00.499067+0000","last_fullsized":"2024-03-30T13:20:00.499067+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:19:48.759444+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:19:48.759444+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:14.394504+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":391,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-03-30T13:20:00.338529+0000","last_change":"2024-03-30T13:19:33.837451+0000","last_active":"2024-03-30T13:20:00.338529+0000","last_peered":"2024-03-30T13:20:00.338529+0000","last_clean":"2024-03-30T13:20:00.338529+0000","last_became_active":"2024-03-30T13:18:32.415413+0000","last_became_peered":"2024-03-30T13:18:32.415413+0000","last_unstale":"2024-03-30T13:20:00.338529+0000","last_undegraded":"2024-03-30T13:20:00.338529+0000","last_fullsized":"2024-03-30T13:20:00.338529+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:19:33.837082+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:19:33.837082+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:54.376722+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":744,"reported_epoch":673,"state":"active+clean","last_fresh":"2024-03-30T13:20:00.338839+0000","last_change":"2024-03-30T13:19:19.587739+0000","last_active":"2024-03-30T13:20:00.338839+0000","last_peered":"2024-03-30T13:20:00.338839+0000","last_clean":"2024-03-30T13:20:00.338839+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:20:00.338839+0000","last_undegraded":"2024-03-30T13:20:00.338839+0000","last_fullsized":"2024-03-30T13:20:00.338839+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:19:19.587471+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:19:19.587471+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:40.749837+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":277446227,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":109256704,"data_stored":109256704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":629,"seq":2701534429204,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59872,"kb_used_data":23124,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94311968,"statfs":{"total":96636764160,"available":96575455232,"internally_reserved":0,"allocated":23678976,"data_stored":19173751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157507,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65524,"kb_used_data":27880,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94306316,"statfs":{"total":96636764160,"available":96569667584,"internally_reserved":0,"allocated":28549120,"data_stored":24037136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509738,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56448,"kb_used_data":21108,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315392,"statfs":{"total":96636764160,"available":96578961408,"internally_reserved":0,"allocated":21614592,"data_stored":17102608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375999017,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51916,"kb_used_data":14720,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94319924,"statfs":{"total":96636764160,"available":96583602176,"internally_reserved":0,"allocated":15073280,"data_stored":10561296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509739,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57368,"kb_used_data":22032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94314472,"statfs":{"total":96636764160,"available":96578019328,"internally_reserved":0,"allocated":22560768,"data_stored":18048784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742410,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54500,"kb_used_data":18072,"kb_used_omap":15,"kb_used_meta":36400,"kb_avail":94317340,"statfs":{"total":96636764160,"available":96580956160,"internally_reserved":0,"allocated":18505728,"data_stored":13993744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":37274051},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147448,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56080,"kb_used_data":24388,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94315760,"statfs":{"total":96636764160,"available":96579338240,"internally_reserved":0,"allocated":24973312,"data_stored":20461328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14340096,"data_stored":14340096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6852608,"data_stored":6852608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13393920,"data_stored":13393920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16752640,"data_stored":16752640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:20:07.240 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:20:07.240 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:20:07.240 INFO:tasks.thrashosds.thrasher:Reviving osd 4 2024-03-30T13:20:07.240 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-30T13:20:07.241 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-30T13:20:07.243 INFO:tasks.ceph.osd.4:Started 2024-03-30T13:20:07.244 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-30T13:20:07.313 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:20:07.314+0000 7f31bbea97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:07.320 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:20:07.322+0000 7f31bbea97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:07.320 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:20:07.322+0000 7f31bbea97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:07.361 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T13:20:07.361 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T13:20:07.361 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T13:20:07.361 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T13:20:07.361 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T13:20:07.361 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T13:20:07.361 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-03-30T13:20:07.361 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-03-30T13:20:07.361 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-03-30T13:20:07.361 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-03-30T13:20:07.361 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-03-30T13:20:07.361 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T13:20:07.363 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:20:07.363 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-03-30T13:20:07.861 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:20:07.862+0000 7f31bbea97c0 -1 Falling back to public interface 2024-03-30T13:20:11.677 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:20:11.678+0000 7f31bbea97c0 -1 osd.4 666 log_to_monitors true 2024-03-30T13:20:12.365 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-30T13:20:12.452 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:20:12.454+0000 7f31afc76640 -1 osd.4 666 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:20:12.500 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T13:20:12.500 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T13:20:12.500 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T13:20:12.500 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T13:20:12.514 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:20:12.514 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-03-30T13:20:12.514 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 4 2024-03-30T13:20:12.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:12.642+0000 7f308f5bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:12.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:12.658+0000 7f308f5bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:12.839 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:20:12.838+0000 7fbb38f08640 -1 mon.a@0(leader).osd e676 definitely_dead 0 2024-03-30T13:20:13.415 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.4. 2024-03-30T13:20:13.443 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-03-30T13:20:13.443 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:20:13.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:13.562+0000 7fb5908e0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:13.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:13.578+0000 7fb5908e0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:14.417 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:20:14.446 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:20:14.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:14.558+0000 7ff785128640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:14.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:14.574+0000 7ff785128640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:15.439 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:20:15.464 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:20:15.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:15.582+0000 7f4fa4a90640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:15.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:15.598+0000 7f4fa4a90640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:16.451 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:20:16.477 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:20:16.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:16.598+0000 7ffa7f511640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:16.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:16.614+0000 7ffa7f511640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:17.463 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:20:17.489 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:20:17.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:17.606+0000 7fe991fec640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:17.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:17.622+0000 7fe991fec640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:18.477 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:20:18.502 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:20:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:18.622+0000 7ff32971c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:18.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:18.638+0000 7ff32971c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:19.499 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:20:19.522 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:20:19.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:19.642+0000 7f29f2101640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:19.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:19.662+0000 7f29f2101640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:20.508 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:20:20.530 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:20:20.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:20.646+0000 7f9af3963640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:20.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:20.662+0000 7f9af3963640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:21.503 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:20:21.531 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:20:21.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:21.650+0000 7ff2c6ef9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:21.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:21.666+0000 7ff2c6ef9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:22.524 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:20:22.549 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:20:22.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:22.666+0000 7f704ced1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:22.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:22.682+0000 7f704ced1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:23.534 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:20:23.560 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:20:23.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:23.682+0000 7eff64ed2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:23.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:23.698+0000 7eff64ed2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:24.542 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:20:24.566 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:20:24.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:24.682+0000 7f96b731a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:24.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:24.698+0000 7f96b731a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:25.557 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:20:25.582 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:20:25.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:25.698+0000 7f8f0d499640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:25.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:25.718+0000 7f8f0d499640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:26.576 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:20:26.599 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:20:26.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:26.710+0000 7ff6821bc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:26.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:26.726+0000 7ff6821bc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:27.577 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:20:27.603 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:20:27.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:27.726+0000 7f26ee017640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:27.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:27.742+0000 7f26ee017640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:28.580 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:20:28.609 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:20:28.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:28.730+0000 7f75f8c29640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:28.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:28.746+0000 7f75f8c29640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:29.583 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:20:29.612 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:20:29.612 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:20:29.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:29.726+0000 7f9b33a00640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:29.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:29.738+0000 7f9b33a00640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:29.903 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:20:29.904 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:20:29.919 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1434,"stamp":"2024-03-30T13:20:29.588321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":277446227,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":448260,"kb_used_data":160184,"kb_used_omap":64,"kb_used_meta":287871,"kb_avail":754526460,"statfs":{"total":773094113280,"available":772635095040,"internally_reserved":0,"allocated":164028416,"data_stored":127109365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":65845,"internal_metadata":294780619},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.026033"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1610,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.103118+0000","last_change":"2024-03-30T13:20:14.433776+0000","last_active":"2024-03-30T13:20:26.103118+0000","last_peered":"2024-03-30T13:20:26.103118+0000","last_clean":"2024-03-30T13:20:26.103118+0000","last_became_active":"2024-03-30T13:20:14.431136+0000","last_became_peered":"2024-03-30T13:20:14.431136+0000","last_unstale":"2024-03-30T13:20:26.103118+0000","last_undegraded":"2024-03-30T13:20:26.103118+0000","last_fullsized":"2024-03-30T13:20:26.103118+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:02.484916+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:20:02.484916+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:25.338815+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1443,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-03-30T13:20:27.855957+0000","last_change":"2024-03-30T13:20:14.428285+0000","last_active":"2024-03-30T13:20:27.855957+0000","last_peered":"2024-03-30T13:20:27.855957+0000","last_clean":"2024-03-30T13:20:27.855957+0000","last_became_active":"2024-03-30T13:20:14.427294+0000","last_became_peered":"2024-03-30T13:20:14.427294+0000","last_unstale":"2024-03-30T13:20:27.855957+0000","last_undegraded":"2024-03-30T13:20:27.855957+0000","last_fullsized":"2024-03-30T13:20:27.855957+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:19:43.319043+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:19:43.319043+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:43.482661+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":219,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.741057+0000","last_change":"2024-03-30T13:20:14.423578+0000","last_active":"2024-03-30T13:20:26.741057+0000","last_peered":"2024-03-30T13:20:26.741057+0000","last_clean":"2024-03-30T13:20:26.741057+0000","last_became_active":"2024-03-30T13:20:14.422234+0000","last_became_peered":"2024-03-30T13:20:14.422234+0000","last_unstale":"2024-03-30T13:20:26.741057+0000","last_undegraded":"2024-03-30T13:20:26.741057+0000","last_fullsized":"2024-03-30T13:20:26.741057+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:19:30.741663+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:19:30.741663+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:38.004904+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":215,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.740991+0000","last_change":"2024-03-30T13:20:10.775281+0000","last_active":"2024-03-30T13:20:26.740991+0000","last_peered":"2024-03-30T13:20:26.740991+0000","last_clean":"2024-03-30T13:20:26.740991+0000","last_became_active":"2024-03-30T13:19:23.949921+0000","last_became_peered":"2024-03-30T13:19:23.949921+0000","last_unstale":"2024-03-30T13:20:26.740991+0000","last_undegraded":"2024-03-30T13:20:26.740991+0000","last_fullsized":"2024-03-30T13:20:26.740991+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:10.775032+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:20:10.775032+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:18.932592+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":412,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.740346+0000","last_change":"2024-03-30T13:20:25.690889+0000","last_active":"2024-03-30T13:20:26.740346+0000","last_peered":"2024-03-30T13:20:26.740346+0000","last_clean":"2024-03-30T13:20:26.740346+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:20:26.740346+0000","last_undegraded":"2024-03-30T13:20:26.740346+0000","last_fullsized":"2024-03-30T13:20:26.740346+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:20:25.690651+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:55.652556+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1574,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.737553+0000","last_change":"2024-03-30T13:20:04.494258+0000","last_active":"2024-03-30T13:20:26.737553+0000","last_peered":"2024-03-30T13:20:26.737553+0000","last_clean":"2024-03-30T13:20:26.737553+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:20:26.737553+0000","last_undegraded":"2024-03-30T13:20:26.737553+0000","last_fullsized":"2024-03-30T13:20:26.737553+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:20:04.493986+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_clean_scrub_stamp":"2024-03-30T13:20:04.493986+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:09.894871+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":267,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.741061+0000","last_change":"2024-03-30T13:20:14.423014+0000","last_active":"2024-03-30T13:20:26.741061+0000","last_peered":"2024-03-30T13:20:26.741061+0000","last_clean":"2024-03-30T13:20:26.741061+0000","last_became_active":"2024-03-30T13:20:14.421760+0000","last_became_peered":"2024-03-30T13:20:14.421760+0000","last_unstale":"2024-03-30T13:20:26.741061+0000","last_undegraded":"2024-03-30T13:20:26.741061+0000","last_fullsized":"2024-03-30T13:20:26.741061+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:19:35.780073+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:19:35.780073+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:41.822890+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":818,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-03-30T13:20:24.340741+0000","last_change":"2024-03-30T13:20:23.372796+0000","last_active":"2024-03-30T13:20:24.340741+0000","last_peered":"2024-03-30T13:20:24.340741+0000","last_clean":"2024-03-30T13:20:24.340741+0000","last_became_active":"2024-03-30T13:18:32.413904+0000","last_became_peered":"2024-03-30T13:18:32.413904+0000","last_unstale":"2024-03-30T13:20:24.340741+0000","last_undegraded":"2024-03-30T13:20:24.340741+0000","last_fullsized":"2024-03-30T13:20:24.340741+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:20:23.372460+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:20:23.372460+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:29.313113+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1729,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.737512+0000","last_change":"2024-03-30T13:19:37.564272+0000","last_active":"2024-03-30T13:20:26.737512+0000","last_peered":"2024-03-30T13:20:26.737512+0000","last_clean":"2024-03-30T13:20:26.737512+0000","last_became_active":"2024-03-30T13:19:23.380536+0000","last_became_peered":"2024-03-30T13:19:23.380536+0000","last_unstale":"2024-03-30T13:20:26.737512+0000","last_undegraded":"2024-03-30T13:20:26.737512+0000","last_fullsized":"2024-03-30T13:20:26.737512+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:19:37.564154+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:19:37.564154+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:51.343381+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":86286016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1288,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-03-30T13:20:27.855804+0000","last_change":"2024-03-30T13:20:23.631457+0000","last_active":"2024-03-30T13:20:27.855804+0000","last_peered":"2024-03-30T13:20:27.855804+0000","last_clean":"2024-03-30T13:20:27.855804+0000","last_became_active":"2024-03-30T13:20:14.427056+0000","last_became_peered":"2024-03-30T13:20:14.427056+0000","last_unstale":"2024-03-30T13:20:27.855804+0000","last_undegraded":"2024-03-30T13:20:27.855804+0000","last_fullsized":"2024-03-30T13:20:27.855804+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:20:23.631067+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:42.480355+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1585,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.103012+0000","last_change":"2024-03-30T13:20:14.433762+0000","last_active":"2024-03-30T13:20:26.103012+0000","last_peered":"2024-03-30T13:20:26.103012+0000","last_clean":"2024-03-30T13:20:26.103012+0000","last_became_active":"2024-03-30T13:20:14.431380+0000","last_became_peered":"2024-03-30T13:20:14.431380+0000","last_unstale":"2024-03-30T13:20:26.103012+0000","last_undegraded":"2024-03-30T13:20:26.103012+0000","last_fullsized":"2024-03-30T13:20:26.103012+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:19:19.392102+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_clean_scrub_stamp":"2024-03-30T13:19:19.392102+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:41.944162+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1613,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.103042+0000","last_change":"2024-03-30T13:19:05.199227+0000","last_active":"2024-03-30T13:20:26.103042+0000","last_peered":"2024-03-30T13:20:26.103042+0000","last_clean":"2024-03-30T13:20:26.103042+0000","last_became_active":"2024-03-30T13:18:32.419902+0000","last_became_peered":"2024-03-30T13:18:32.419902+0000","last_unstale":"2024-03-30T13:20:26.103042+0000","last_undegraded":"2024-03-30T13:20:26.103042+0000","last_fullsized":"2024-03-30T13:20:26.103042+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:19:05.198795+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:19:05.198795+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:32.312402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":417,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.740315+0000","last_change":"2024-03-30T13:19:00.934425+0000","last_active":"2024-03-30T13:20:26.740315+0000","last_peered":"2024-03-30T13:20:26.740315+0000","last_clean":"2024-03-30T13:20:26.740315+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:20:26.740315+0000","last_undegraded":"2024-03-30T13:20:26.740315+0000","last_fullsized":"2024-03-30T13:20:26.740315+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:19:00.934164+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:19:00.934164+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":206,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.741019+0000","last_change":"2024-03-30T13:19:48.759695+0000","last_active":"2024-03-30T13:20:26.741019+0000","last_peered":"2024-03-30T13:20:26.741019+0000","last_clean":"2024-03-30T13:20:26.741019+0000","last_became_active":"2024-03-30T13:19:23.380857+0000","last_became_peered":"2024-03-30T13:19:23.380857+0000","last_unstale":"2024-03-30T13:20:26.741019+0000","last_undegraded":"2024-03-30T13:20:26.741019+0000","last_fullsized":"2024-03-30T13:20:26.741019+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:19:48.759444+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:19:48.759444+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:14.394504+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":423,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.740359+0000","last_change":"2024-03-30T13:19:33.837451+0000","last_active":"2024-03-30T13:20:26.740359+0000","last_peered":"2024-03-30T13:20:26.740359+0000","last_clean":"2024-03-30T13:20:26.740359+0000","last_became_active":"2024-03-30T13:18:32.415413+0000","last_became_peered":"2024-03-30T13:18:32.415413+0000","last_unstale":"2024-03-30T13:20:26.740359+0000","last_undegraded":"2024-03-30T13:20:26.740359+0000","last_fullsized":"2024-03-30T13:20:26.740359+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:19:33.837082+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:19:33.837082+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:54.376722+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":771,"reported_epoch":687,"state":"active+clean","last_fresh":"2024-03-30T13:20:24.340743+0000","last_change":"2024-03-30T13:19:19.587739+0000","last_active":"2024-03-30T13:20:24.340743+0000","last_peered":"2024-03-30T13:20:24.340743+0000","last_clean":"2024-03-30T13:20:24.340743+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:20:24.340743+0000","last_undegraded":"2024-03-30T13:20:24.340743+0000","last_fullsized":"2024-03-30T13:20:24.340743+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:19:19.587471+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:19:19.587471+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:40.749837+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":277446227,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":629,"seq":2701534429209,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60064,"kb_used_data":23316,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94311776,"statfs":{"total":96636764160,"available":96575258624,"internally_reserved":0,"allocated":23875584,"data_stored":19263228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157512,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65720,"kb_used_data":28072,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94306120,"statfs":{"total":96636764160,"available":96569466880,"internally_reserved":0,"allocated":28745728,"data_stored":24126621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":677,"seq":2907692859397,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57440,"kb_used_data":19788,"kb_used_omap":13,"kb_used_meta":37618,"kb_avail":94314400,"statfs":{"total":96636764160,"available":96577945600,"internally_reserved":0,"allocated":20262912,"data_stored":15637004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14287,"internal_metadata":38520881},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509742,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49860,"kb_used_data":14520,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94321980,"statfs":{"total":96636764160,"available":96585707520,"internally_reserved":0,"allocated":14868480,"data_stored":10262925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375999021,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50996,"kb_used_data":13796,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94320844,"statfs":{"total":96636764160,"available":96584544256,"internally_reserved":0,"allocated":14127104,"data_stored":9521549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509743,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57044,"kb_used_data":21696,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94314796,"statfs":{"total":96636764160,"available":96578351104,"internally_reserved":0,"allocated":22216704,"data_stored":17611149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742415,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54716,"kb_used_data":18276,"kb_used_omap":15,"kb_used_meta":36400,"kb_avail":94317124,"statfs":{"total":96636764160,"available":96580734976,"internally_reserved":0,"allocated":18714624,"data_stored":14088716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":37274051},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147453,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52420,"kb_used_data":20720,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94319420,"statfs":{"total":96636764160,"available":96583086080,"internally_reserved":0,"allocated":21217280,"data_stored":16598173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:20:29.921 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:20:30.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:30.034+0000 7f2ccf955640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:30.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:30.050+0000 7f2ccf955640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:30.220 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:20:30.221 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:20:30.238 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1435,"stamp":"2024-03-30T13:20:30.216257+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":277446227,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":448452,"kb_used_data":160376,"kb_used_omap":64,"kb_used_meta":287871,"kb_avail":754526268,"statfs":{"total":773094113280,"available":772634898432,"internally_reserved":0,"allocated":164225024,"data_stored":127197307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":65845,"internal_metadata":294780619},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001070"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1610,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.103118+0000","last_change":"2024-03-30T13:20:14.433776+0000","last_active":"2024-03-30T13:20:26.103118+0000","last_peered":"2024-03-30T13:20:26.103118+0000","last_clean":"2024-03-30T13:20:26.103118+0000","last_became_active":"2024-03-30T13:20:14.431136+0000","last_became_peered":"2024-03-30T13:20:14.431136+0000","last_unstale":"2024-03-30T13:20:26.103118+0000","last_undegraded":"2024-03-30T13:20:26.103118+0000","last_fullsized":"2024-03-30T13:20:26.103118+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:02.484916+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:20:02.484916+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:25.338815+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1443,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-03-30T13:20:27.855957+0000","last_change":"2024-03-30T13:20:14.428285+0000","last_active":"2024-03-30T13:20:27.855957+0000","last_peered":"2024-03-30T13:20:27.855957+0000","last_clean":"2024-03-30T13:20:27.855957+0000","last_became_active":"2024-03-30T13:20:14.427294+0000","last_became_peered":"2024-03-30T13:20:14.427294+0000","last_unstale":"2024-03-30T13:20:27.855957+0000","last_undegraded":"2024-03-30T13:20:27.855957+0000","last_fullsized":"2024-03-30T13:20:27.855957+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:19:43.319043+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:19:43.319043+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:43.482661+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":219,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.741057+0000","last_change":"2024-03-30T13:20:14.423578+0000","last_active":"2024-03-30T13:20:26.741057+0000","last_peered":"2024-03-30T13:20:26.741057+0000","last_clean":"2024-03-30T13:20:26.741057+0000","last_became_active":"2024-03-30T13:20:14.422234+0000","last_became_peered":"2024-03-30T13:20:14.422234+0000","last_unstale":"2024-03-30T13:20:26.741057+0000","last_undegraded":"2024-03-30T13:20:26.741057+0000","last_fullsized":"2024-03-30T13:20:26.741057+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:19:30.741663+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:19:30.741663+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:38.004904+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":215,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.740991+0000","last_change":"2024-03-30T13:20:10.775281+0000","last_active":"2024-03-30T13:20:26.740991+0000","last_peered":"2024-03-30T13:20:26.740991+0000","last_clean":"2024-03-30T13:20:26.740991+0000","last_became_active":"2024-03-30T13:19:23.949921+0000","last_became_peered":"2024-03-30T13:19:23.949921+0000","last_unstale":"2024-03-30T13:20:26.740991+0000","last_undegraded":"2024-03-30T13:20:26.740991+0000","last_fullsized":"2024-03-30T13:20:26.740991+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:10.775032+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:20:10.775032+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:18.932592+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":412,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.740346+0000","last_change":"2024-03-30T13:20:25.690889+0000","last_active":"2024-03-30T13:20:26.740346+0000","last_peered":"2024-03-30T13:20:26.740346+0000","last_clean":"2024-03-30T13:20:26.740346+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:20:26.740346+0000","last_undegraded":"2024-03-30T13:20:26.740346+0000","last_fullsized":"2024-03-30T13:20:26.740346+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:20:25.690651+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:55.652556+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1574,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.737553+0000","last_change":"2024-03-30T13:20:04.494258+0000","last_active":"2024-03-30T13:20:26.737553+0000","last_peered":"2024-03-30T13:20:26.737553+0000","last_clean":"2024-03-30T13:20:26.737553+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:20:26.737553+0000","last_undegraded":"2024-03-30T13:20:26.737553+0000","last_fullsized":"2024-03-30T13:20:26.737553+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:20:04.493986+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_clean_scrub_stamp":"2024-03-30T13:20:04.493986+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:09.894871+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":267,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.741061+0000","last_change":"2024-03-30T13:20:14.423014+0000","last_active":"2024-03-30T13:20:26.741061+0000","last_peered":"2024-03-30T13:20:26.741061+0000","last_clean":"2024-03-30T13:20:26.741061+0000","last_became_active":"2024-03-30T13:20:14.421760+0000","last_became_peered":"2024-03-30T13:20:14.421760+0000","last_unstale":"2024-03-30T13:20:26.741061+0000","last_undegraded":"2024-03-30T13:20:26.741061+0000","last_fullsized":"2024-03-30T13:20:26.741061+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:19:35.780073+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:19:35.780073+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:41.822890+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":829,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-03-30T13:20:29.638697+0000","last_change":"2024-03-30T13:20:23.372796+0000","last_active":"2024-03-30T13:20:29.638697+0000","last_peered":"2024-03-30T13:20:29.638697+0000","last_clean":"2024-03-30T13:20:29.638697+0000","last_became_active":"2024-03-30T13:18:32.413904+0000","last_became_peered":"2024-03-30T13:18:32.413904+0000","last_unstale":"2024-03-30T13:20:29.638697+0000","last_undegraded":"2024-03-30T13:20:29.638697+0000","last_fullsized":"2024-03-30T13:20:29.638697+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:20:23.372460+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:20:23.372460+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:29.313113+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1729,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.737512+0000","last_change":"2024-03-30T13:19:37.564272+0000","last_active":"2024-03-30T13:20:26.737512+0000","last_peered":"2024-03-30T13:20:26.737512+0000","last_clean":"2024-03-30T13:20:26.737512+0000","last_became_active":"2024-03-30T13:19:23.380536+0000","last_became_peered":"2024-03-30T13:19:23.380536+0000","last_unstale":"2024-03-30T13:20:26.737512+0000","last_undegraded":"2024-03-30T13:20:26.737512+0000","last_fullsized":"2024-03-30T13:20:26.737512+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:19:37.564154+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:19:37.564154+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:51.343381+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":86286016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1288,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-03-30T13:20:27.855804+0000","last_change":"2024-03-30T13:20:23.631457+0000","last_active":"2024-03-30T13:20:27.855804+0000","last_peered":"2024-03-30T13:20:27.855804+0000","last_clean":"2024-03-30T13:20:27.855804+0000","last_became_active":"2024-03-30T13:20:14.427056+0000","last_became_peered":"2024-03-30T13:20:14.427056+0000","last_unstale":"2024-03-30T13:20:27.855804+0000","last_undegraded":"2024-03-30T13:20:27.855804+0000","last_fullsized":"2024-03-30T13:20:27.855804+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:20:23.631067+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:42.480355+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1585,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.103012+0000","last_change":"2024-03-30T13:20:14.433762+0000","last_active":"2024-03-30T13:20:26.103012+0000","last_peered":"2024-03-30T13:20:26.103012+0000","last_clean":"2024-03-30T13:20:26.103012+0000","last_became_active":"2024-03-30T13:20:14.431380+0000","last_became_peered":"2024-03-30T13:20:14.431380+0000","last_unstale":"2024-03-30T13:20:26.103012+0000","last_undegraded":"2024-03-30T13:20:26.103012+0000","last_fullsized":"2024-03-30T13:20:26.103012+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:19:19.392102+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_clean_scrub_stamp":"2024-03-30T13:19:19.392102+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:41.944162+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1613,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.103042+0000","last_change":"2024-03-30T13:19:05.199227+0000","last_active":"2024-03-30T13:20:26.103042+0000","last_peered":"2024-03-30T13:20:26.103042+0000","last_clean":"2024-03-30T13:20:26.103042+0000","last_became_active":"2024-03-30T13:18:32.419902+0000","last_became_peered":"2024-03-30T13:18:32.419902+0000","last_unstale":"2024-03-30T13:20:26.103042+0000","last_undegraded":"2024-03-30T13:20:26.103042+0000","last_fullsized":"2024-03-30T13:20:26.103042+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:19:05.198795+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:19:05.198795+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:32.312402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":417,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.740315+0000","last_change":"2024-03-30T13:19:00.934425+0000","last_active":"2024-03-30T13:20:26.740315+0000","last_peered":"2024-03-30T13:20:26.740315+0000","last_clean":"2024-03-30T13:20:26.740315+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:20:26.740315+0000","last_undegraded":"2024-03-30T13:20:26.740315+0000","last_fullsized":"2024-03-30T13:20:26.740315+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:19:00.934164+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:19:00.934164+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":206,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.741019+0000","last_change":"2024-03-30T13:19:48.759695+0000","last_active":"2024-03-30T13:20:26.741019+0000","last_peered":"2024-03-30T13:20:26.741019+0000","last_clean":"2024-03-30T13:20:26.741019+0000","last_became_active":"2024-03-30T13:19:23.380857+0000","last_became_peered":"2024-03-30T13:19:23.380857+0000","last_unstale":"2024-03-30T13:20:26.741019+0000","last_undegraded":"2024-03-30T13:20:26.741019+0000","last_fullsized":"2024-03-30T13:20:26.741019+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:19:48.759444+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:19:48.759444+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:14.394504+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":423,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.740359+0000","last_change":"2024-03-30T13:19:33.837451+0000","last_active":"2024-03-30T13:20:26.740359+0000","last_peered":"2024-03-30T13:20:26.740359+0000","last_clean":"2024-03-30T13:20:26.740359+0000","last_became_active":"2024-03-30T13:18:32.415413+0000","last_became_peered":"2024-03-30T13:18:32.415413+0000","last_unstale":"2024-03-30T13:20:26.740359+0000","last_undegraded":"2024-03-30T13:20:26.740359+0000","last_fullsized":"2024-03-30T13:20:26.740359+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:19:33.837082+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:19:33.837082+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:54.376722+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":782,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-03-30T13:20:29.638910+0000","last_change":"2024-03-30T13:19:19.587739+0000","last_active":"2024-03-30T13:20:29.638910+0000","last_peered":"2024-03-30T13:20:29.638910+0000","last_clean":"2024-03-30T13:20:29.638910+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:20:29.638910+0000","last_undegraded":"2024-03-30T13:20:29.638910+0000","last_fullsized":"2024-03-30T13:20:29.638910+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:19:19.587471+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:19:19.587471+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:40.749837+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":277446227,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":629,"seq":2701534429209,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60064,"kb_used_data":23316,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94311776,"statfs":{"total":96636764160,"available":96575258624,"internally_reserved":0,"allocated":23875584,"data_stored":19263228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157512,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65720,"kb_used_data":28072,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94306120,"statfs":{"total":96636764160,"available":96569466880,"internally_reserved":0,"allocated":28745728,"data_stored":24126621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":677,"seq":2907692859397,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57440,"kb_used_data":19788,"kb_used_omap":13,"kb_used_meta":37618,"kb_avail":94314400,"statfs":{"total":96636764160,"available":96577945600,"internally_reserved":0,"allocated":20262912,"data_stored":15637004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14287,"internal_metadata":38520881},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509743,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49928,"kb_used_data":14580,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94321912,"statfs":{"total":96636764160,"available":96585637888,"internally_reserved":0,"allocated":14929920,"data_stored":10290410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375999022,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51052,"kb_used_data":13856,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94320788,"statfs":{"total":96636764160,"available":96584486912,"internally_reserved":0,"allocated":14188544,"data_stored":9549034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509744,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57112,"kb_used_data":21768,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94314728,"statfs":{"total":96636764160,"available":96578281472,"internally_reserved":0,"allocated":22290432,"data_stored":17644121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742415,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54716,"kb_used_data":18276,"kb_used_omap":15,"kb_used_meta":36400,"kb_avail":94317124,"statfs":{"total":96636764160,"available":96580734976,"internally_reserved":0,"allocated":18714624,"data_stored":14088716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":37274051},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147453,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52420,"kb_used_data":20720,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94319420,"statfs":{"total":96636764160,"available":96583086080,"internally_reserved":0,"allocated":21217280,"data_stored":16598173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:20:30.239 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:20:30.240 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:20:30.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:30.350+0000 7f3346627640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:30.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:30.362+0000 7f3346627640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:30.542 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:20:30.542 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":693,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:20:29.582329+0000","last_up_change":"2024-03-30T13:20:13.403506+0000","last_in_change":"2024-03-30T13:20:13.403506+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":34,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":668,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":461,"last_clean_end":575,"up_from":581,"up_thru":646,"down_at":576,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6818","nonce":4030404750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6821","nonce":4030404750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6828","nonce":4030404750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6825","nonce":4030404750}]},"public_addr":"172.21.15.67:6818/4030404750","cluster_addr":"172.21.15.67:6821/4030404750","heartbeat_back_addr":"172.21.15.67:6828/4030404750","heartbeat_front_addr":"172.21.15.67:6825/4030404750","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":668,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":388,"up_from":393,"up_thru":650,"down_at":389,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6801","nonce":3837194718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6803","nonce":3837194718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6807","nonce":3837194718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3837194718},{"type":"v1","addr":"172.21.15.67:6805","nonce":3837194718}]},"public_addr":"172.21.15.67:6801/3837194718","cluster_addr":"172.21.15.67:6803/3837194718","heartbeat_back_addr":"172.21.15.67:6807/3837194718","heartbeat_front_addr":"172.21.15.67:6805/3837194718","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":559,"last_clean_end":666,"up_from":677,"up_thru":677,"down_at":667,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1606011629},{"type":"v1","addr":"172.21.15.82:6803","nonce":1606011629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":1606011629},{"type":"v1","addr":"172.21.15.82:6807","nonce":1606011629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":1606011629},{"type":"v1","addr":"172.21.15.82:6815","nonce":1606011629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":1606011629},{"type":"v1","addr":"172.21.15.82:6811","nonce":1606011629}]},"public_addr":"172.21.15.82:6803/1606011629","cluster_addr":"172.21.15.82:6807/1606011629","heartbeat_back_addr":"172.21.15.82:6815/1606011629","heartbeat_front_addr":"172.21.15.82:6811/1606011629","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":488,"last_clean_end":506,"up_from":515,"up_thru":646,"down_at":507,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6817","nonce":292505453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6819","nonce":292505453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6823","nonce":292505453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6821","nonce":292505453}]},"public_addr":"172.21.15.82:6817/292505453","cluster_addr":"172.21.15.82:6819/292505453","heartbeat_back_addr":"172.21.15.82:6823/292505453","heartbeat_front_addr":"172.21.15.82:6821/292505453","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":372,"last_clean_end":621,"up_from":629,"up_thru":677,"down_at":622,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3888194455},{"type":"v1","addr":"172.21.15.82:6801","nonce":3888194455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3888194455},{"type":"v1","addr":"172.21.15.82:6806","nonce":3888194455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3888194455},{"type":"v1","addr":"172.21.15.82:6814","nonce":3888194455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3888194455},{"type":"v1","addr":"172.21.15.82:6810","nonce":3888194455}]},"public_addr":"172.21.15.82:6801/3888194455","cluster_addr":"172.21.15.82:6806/3888194455","heartbeat_back_addr":"172.21.15.82:6814/3888194455","heartbeat_front_addr":"172.21.15.82:6810/3888194455","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":604,"last_clean_end":645,"up_from":650,"up_thru":677,"down_at":646,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6825","nonce":1497758647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6827","nonce":1497758647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6831","nonce":1497758647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6829","nonce":1497758647}]},"public_addr":"172.21.15.82:6825/1497758647","cluster_addr":"172.21.15.82:6827/1497758647","heartbeat_back_addr":"172.21.15.82:6831/1497758647","heartbeat_front_addr":"172.21.15.82:6829/1497758647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:20:04.387532+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:20:08.907414+0000","dead_epoch":576},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:19:44.494493+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:19:51.421762+0000","dead_epoch":389},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:20:24.611629+0000","dead_epoch":667},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:20:00.279102+0000","dead_epoch":507},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:19:55.088122+0000","dead_epoch":622},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:19:21.682043+0000","dead_epoch":646}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:20:30.562 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:20:30.562 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:20:30.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:30.670+0000 7f61538b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:30.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:30.686+0000 7f61538b8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:30.862 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:20:30.880 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:20:30.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:30.986+0000 7ff355f76640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:31.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:31.002+0000 7ff355f76640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:31.171 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:20:31.172 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:20:31.189 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:20:31.189 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:20:31.189 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:20:31.190 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:20:31.190 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 3, 5, 1, 6, 7, 4] 2024-03-30T13:20:31.190 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:20:31.190 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:20:31.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:31.294+0000 7faca79af640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:31.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:31.310+0000 7faca79af640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:31.473 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:20:31.473 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:20:31.490 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1435,"stamp":"2024-03-30T13:20:30.216257+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":277446227,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":448452,"kb_used_data":160376,"kb_used_omap":64,"kb_used_meta":287871,"kb_avail":754526268,"statfs":{"total":773094113280,"available":772634898432,"internally_reserved":0,"allocated":164225024,"data_stored":127197307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":65845,"internal_metadata":294780619},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001070"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1610,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.103118+0000","last_change":"2024-03-30T13:20:14.433776+0000","last_active":"2024-03-30T13:20:26.103118+0000","last_peered":"2024-03-30T13:20:26.103118+0000","last_clean":"2024-03-30T13:20:26.103118+0000","last_became_active":"2024-03-30T13:20:14.431136+0000","last_became_peered":"2024-03-30T13:20:14.431136+0000","last_unstale":"2024-03-30T13:20:26.103118+0000","last_undegraded":"2024-03-30T13:20:26.103118+0000","last_fullsized":"2024-03-30T13:20:26.103118+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:02.484916+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:20:02.484916+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:25.338815+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1443,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-03-30T13:20:27.855957+0000","last_change":"2024-03-30T13:20:14.428285+0000","last_active":"2024-03-30T13:20:27.855957+0000","last_peered":"2024-03-30T13:20:27.855957+0000","last_clean":"2024-03-30T13:20:27.855957+0000","last_became_active":"2024-03-30T13:20:14.427294+0000","last_became_peered":"2024-03-30T13:20:14.427294+0000","last_unstale":"2024-03-30T13:20:27.855957+0000","last_undegraded":"2024-03-30T13:20:27.855957+0000","last_fullsized":"2024-03-30T13:20:27.855957+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:19:43.319043+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:19:43.319043+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:43.482661+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":219,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.741057+0000","last_change":"2024-03-30T13:20:14.423578+0000","last_active":"2024-03-30T13:20:26.741057+0000","last_peered":"2024-03-30T13:20:26.741057+0000","last_clean":"2024-03-30T13:20:26.741057+0000","last_became_active":"2024-03-30T13:20:14.422234+0000","last_became_peered":"2024-03-30T13:20:14.422234+0000","last_unstale":"2024-03-30T13:20:26.741057+0000","last_undegraded":"2024-03-30T13:20:26.741057+0000","last_fullsized":"2024-03-30T13:20:26.741057+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:19:30.741663+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:19:30.741663+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:38.004904+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":215,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.740991+0000","last_change":"2024-03-30T13:20:10.775281+0000","last_active":"2024-03-30T13:20:26.740991+0000","last_peered":"2024-03-30T13:20:26.740991+0000","last_clean":"2024-03-30T13:20:26.740991+0000","last_became_active":"2024-03-30T13:19:23.949921+0000","last_became_peered":"2024-03-30T13:19:23.949921+0000","last_unstale":"2024-03-30T13:20:26.740991+0000","last_undegraded":"2024-03-30T13:20:26.740991+0000","last_fullsized":"2024-03-30T13:20:26.740991+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:10.775032+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:20:10.775032+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:18.932592+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":412,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.740346+0000","last_change":"2024-03-30T13:20:25.690889+0000","last_active":"2024-03-30T13:20:26.740346+0000","last_peered":"2024-03-30T13:20:26.740346+0000","last_clean":"2024-03-30T13:20:26.740346+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:20:26.740346+0000","last_undegraded":"2024-03-30T13:20:26.740346+0000","last_fullsized":"2024-03-30T13:20:26.740346+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:20:25.690651+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:55.652556+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1574,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.737553+0000","last_change":"2024-03-30T13:20:04.494258+0000","last_active":"2024-03-30T13:20:26.737553+0000","last_peered":"2024-03-30T13:20:26.737553+0000","last_clean":"2024-03-30T13:20:26.737553+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:20:26.737553+0000","last_undegraded":"2024-03-30T13:20:26.737553+0000","last_fullsized":"2024-03-30T13:20:26.737553+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:20:04.493986+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_clean_scrub_stamp":"2024-03-30T13:20:04.493986+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:09.894871+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":267,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.741061+0000","last_change":"2024-03-30T13:20:14.423014+0000","last_active":"2024-03-30T13:20:26.741061+0000","last_peered":"2024-03-30T13:20:26.741061+0000","last_clean":"2024-03-30T13:20:26.741061+0000","last_became_active":"2024-03-30T13:20:14.421760+0000","last_became_peered":"2024-03-30T13:20:14.421760+0000","last_unstale":"2024-03-30T13:20:26.741061+0000","last_undegraded":"2024-03-30T13:20:26.741061+0000","last_fullsized":"2024-03-30T13:20:26.741061+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:19:35.780073+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:19:35.780073+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:41.822890+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":829,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-03-30T13:20:29.638697+0000","last_change":"2024-03-30T13:20:23.372796+0000","last_active":"2024-03-30T13:20:29.638697+0000","last_peered":"2024-03-30T13:20:29.638697+0000","last_clean":"2024-03-30T13:20:29.638697+0000","last_became_active":"2024-03-30T13:18:32.413904+0000","last_became_peered":"2024-03-30T13:18:32.413904+0000","last_unstale":"2024-03-30T13:20:29.638697+0000","last_undegraded":"2024-03-30T13:20:29.638697+0000","last_fullsized":"2024-03-30T13:20:29.638697+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:20:23.372460+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:20:23.372460+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:29.313113+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1729,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.737512+0000","last_change":"2024-03-30T13:19:37.564272+0000","last_active":"2024-03-30T13:20:26.737512+0000","last_peered":"2024-03-30T13:20:26.737512+0000","last_clean":"2024-03-30T13:20:26.737512+0000","last_became_active":"2024-03-30T13:19:23.380536+0000","last_became_peered":"2024-03-30T13:19:23.380536+0000","last_unstale":"2024-03-30T13:20:26.737512+0000","last_undegraded":"2024-03-30T13:20:26.737512+0000","last_fullsized":"2024-03-30T13:20:26.737512+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:19:37.564154+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:19:37.564154+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:51.343381+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":86286016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1288,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-03-30T13:20:27.855804+0000","last_change":"2024-03-30T13:20:23.631457+0000","last_active":"2024-03-30T13:20:27.855804+0000","last_peered":"2024-03-30T13:20:27.855804+0000","last_clean":"2024-03-30T13:20:27.855804+0000","last_became_active":"2024-03-30T13:20:14.427056+0000","last_became_peered":"2024-03-30T13:20:14.427056+0000","last_unstale":"2024-03-30T13:20:27.855804+0000","last_undegraded":"2024-03-30T13:20:27.855804+0000","last_fullsized":"2024-03-30T13:20:27.855804+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:20:23.631067+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:42.480355+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1585,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.103012+0000","last_change":"2024-03-30T13:20:14.433762+0000","last_active":"2024-03-30T13:20:26.103012+0000","last_peered":"2024-03-30T13:20:26.103012+0000","last_clean":"2024-03-30T13:20:26.103012+0000","last_became_active":"2024-03-30T13:20:14.431380+0000","last_became_peered":"2024-03-30T13:20:14.431380+0000","last_unstale":"2024-03-30T13:20:26.103012+0000","last_undegraded":"2024-03-30T13:20:26.103012+0000","last_fullsized":"2024-03-30T13:20:26.103012+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:19:19.392102+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_clean_scrub_stamp":"2024-03-30T13:19:19.392102+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:41.944162+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1613,"reported_epoch":689,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.103042+0000","last_change":"2024-03-30T13:19:05.199227+0000","last_active":"2024-03-30T13:20:26.103042+0000","last_peered":"2024-03-30T13:20:26.103042+0000","last_clean":"2024-03-30T13:20:26.103042+0000","last_became_active":"2024-03-30T13:18:32.419902+0000","last_became_peered":"2024-03-30T13:18:32.419902+0000","last_unstale":"2024-03-30T13:20:26.103042+0000","last_undegraded":"2024-03-30T13:20:26.103042+0000","last_fullsized":"2024-03-30T13:20:26.103042+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:19:05.198795+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:19:05.198795+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:32.312402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":417,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.740315+0000","last_change":"2024-03-30T13:19:00.934425+0000","last_active":"2024-03-30T13:20:26.740315+0000","last_peered":"2024-03-30T13:20:26.740315+0000","last_clean":"2024-03-30T13:20:26.740315+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:20:26.740315+0000","last_undegraded":"2024-03-30T13:20:26.740315+0000","last_fullsized":"2024-03-30T13:20:26.740315+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:19:00.934164+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:19:00.934164+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":206,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.741019+0000","last_change":"2024-03-30T13:19:48.759695+0000","last_active":"2024-03-30T13:20:26.741019+0000","last_peered":"2024-03-30T13:20:26.741019+0000","last_clean":"2024-03-30T13:20:26.741019+0000","last_became_active":"2024-03-30T13:19:23.380857+0000","last_became_peered":"2024-03-30T13:19:23.380857+0000","last_unstale":"2024-03-30T13:20:26.741019+0000","last_undegraded":"2024-03-30T13:20:26.741019+0000","last_fullsized":"2024-03-30T13:20:26.741019+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:19:48.759444+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:19:48.759444+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:14.394504+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":423,"reported_epoch":690,"state":"active+clean","last_fresh":"2024-03-30T13:20:26.740359+0000","last_change":"2024-03-30T13:19:33.837451+0000","last_active":"2024-03-30T13:20:26.740359+0000","last_peered":"2024-03-30T13:20:26.740359+0000","last_clean":"2024-03-30T13:20:26.740359+0000","last_became_active":"2024-03-30T13:18:32.415413+0000","last_became_peered":"2024-03-30T13:18:32.415413+0000","last_unstale":"2024-03-30T13:20:26.740359+0000","last_undegraded":"2024-03-30T13:20:26.740359+0000","last_fullsized":"2024-03-30T13:20:26.740359+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:19:33.837082+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:19:33.837082+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:54.376722+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":782,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-03-30T13:20:29.638910+0000","last_change":"2024-03-30T13:19:19.587739+0000","last_active":"2024-03-30T13:20:29.638910+0000","last_peered":"2024-03-30T13:20:29.638910+0000","last_clean":"2024-03-30T13:20:29.638910+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:20:29.638910+0000","last_undegraded":"2024-03-30T13:20:29.638910+0000","last_fullsized":"2024-03-30T13:20:29.638910+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:19:19.587471+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:19:19.587471+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:40.749837+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":277446227,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":629,"seq":2701534429209,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60064,"kb_used_data":23316,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94311776,"statfs":{"total":96636764160,"available":96575258624,"internally_reserved":0,"allocated":23875584,"data_stored":19263228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157512,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65720,"kb_used_data":28072,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94306120,"statfs":{"total":96636764160,"available":96569466880,"internally_reserved":0,"allocated":28745728,"data_stored":24126621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":677,"seq":2907692859397,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57440,"kb_used_data":19788,"kb_used_omap":13,"kb_used_meta":37618,"kb_avail":94314400,"statfs":{"total":96636764160,"available":96577945600,"internally_reserved":0,"allocated":20262912,"data_stored":15637004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14287,"internal_metadata":38520881},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509743,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49928,"kb_used_data":14580,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94321912,"statfs":{"total":96636764160,"available":96585637888,"internally_reserved":0,"allocated":14929920,"data_stored":10290410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375999022,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51052,"kb_used_data":13856,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94320788,"statfs":{"total":96636764160,"available":96584486912,"internally_reserved":0,"allocated":14188544,"data_stored":9549034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509744,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57112,"kb_used_data":21768,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94314728,"statfs":{"total":96636764160,"available":96578281472,"internally_reserved":0,"allocated":22290432,"data_stored":17644121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742415,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54716,"kb_used_data":18276,"kb_used_omap":15,"kb_used_meta":36400,"kb_avail":94317124,"statfs":{"total":96636764160,"available":96580734976,"internally_reserved":0,"allocated":18714624,"data_stored":14088716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":37274051},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":393,"seq":1687922147453,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52420,"kb_used_data":20720,"kb_used_omap":4,"kb_used_meta":31675,"kb_avail":94319420,"statfs":{"total":96636764160,"available":96583086080,"internally_reserved":0,"allocated":21217280,"data_stored":16598173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4398,"internal_metadata":32435922},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:20:31.491 INFO:tasks.thrashosds.thrasher:Choosing PG 1.4 with acting set [3, 5, 7] 2024-03-30T13:20:31.491 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 2, 3, 5, 1, 6, 7, 4] 2024-03-30T13:20:31.491 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-30T13:20:31.491 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:20:31.571 INFO:tasks.ceph.osd.3:Stopped 2024-03-30T13:20:31.571 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 2, 3, 5, 1, 6, 7, 4] 2024-03-30T13:20:31.572 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 3 2024-03-30T13:20:31.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:31.682+0000 7fbf5b52d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:31.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:31.694+0000 7fbf5b52d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:31.866 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:20:31.866+0000 7fbb38f08640 -1 mon.a@0(leader).osd e694 definitely_dead 0 2024-03-30T13:20:32.617 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.3. 2024-03-30T13:20:32.645 INFO:tasks.thrashosds.thrasher:dead_osds=[3], live_osds=[0, 2, 5, 1, 6, 7, 4] 2024-03-30T13:20:32.645 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:20:32.645 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:20:32.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:32.762+0000 7f1db0aae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:32.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:32.774+0000 7f1db0aae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:32.940 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:20:32.941 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:20:32.957 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1438,"stamp":"2024-03-30T13:20:32.622172+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":252067987,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":396192,"kb_used_data":139824,"kb_used_omap":60,"kb_used_meta":256195,"kb_avail":660206688,"statfs":{"total":676457349120,"available":676051648512,"internally_reserved":0,"allocated":143179776,"data_stored":110676354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61447,"internal_metadata":262344697},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-10,"num_bytes_recovered":-25378240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.406460"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1618,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-03-30T13:20:29.642976+0000","last_change":"2024-03-30T13:20:14.433776+0000","last_active":"2024-03-30T13:20:29.642976+0000","last_peered":"2024-03-30T13:20:29.642976+0000","last_clean":"2024-03-30T13:20:29.642976+0000","last_became_active":"2024-03-30T13:20:14.431136+0000","last_became_peered":"2024-03-30T13:20:14.431136+0000","last_unstale":"2024-03-30T13:20:29.642976+0000","last_undegraded":"2024-03-30T13:20:29.642976+0000","last_fullsized":"2024-03-30T13:20:29.642976+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:02.484916+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:20:02.484916+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:25.338815+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1443,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-03-30T13:20:27.855957+0000","last_change":"2024-03-30T13:20:14.428285+0000","last_active":"2024-03-30T13:20:27.855957+0000","last_peered":"2024-03-30T13:20:27.855957+0000","last_clean":"2024-03-30T13:20:27.855957+0000","last_became_active":"2024-03-30T13:20:14.427294+0000","last_became_peered":"2024-03-30T13:20:14.427294+0000","last_unstale":"2024-03-30T13:20:27.855957+0000","last_undegraded":"2024-03-30T13:20:27.855957+0000","last_fullsized":"2024-03-30T13:20:27.855957+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:19:43.319043+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:19:43.319043+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:43.482661+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":227,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-03-30T13:20:31.612569+0000","last_change":"2024-03-30T13:20:14.423578+0000","last_active":"2024-03-30T13:20:31.612569+0000","last_peered":"2024-03-30T13:20:31.612569+0000","last_clean":"2024-03-30T13:20:31.612569+0000","last_became_active":"2024-03-30T13:20:14.422234+0000","last_became_peered":"2024-03-30T13:20:14.422234+0000","last_unstale":"2024-03-30T13:20:31.612569+0000","last_undegraded":"2024-03-30T13:20:31.612569+0000","last_fullsized":"2024-03-30T13:20:31.612569+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:19:30.741663+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:19:30.741663+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:38.004904+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":223,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-03-30T13:20:31.612378+0000","last_change":"2024-03-30T13:20:10.775281+0000","last_active":"2024-03-30T13:20:31.612378+0000","last_peered":"2024-03-30T13:20:31.612378+0000","last_clean":"2024-03-30T13:20:31.612378+0000","last_became_active":"2024-03-30T13:19:23.949921+0000","last_became_peered":"2024-03-30T13:19:23.949921+0000","last_unstale":"2024-03-30T13:20:31.612378+0000","last_undegraded":"2024-03-30T13:20:31.612378+0000","last_fullsized":"2024-03-30T13:20:31.612378+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:10.775032+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:20:10.775032+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:18.932592+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":420,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-03-30T13:20:31.612532+0000","last_change":"2024-03-30T13:20:25.690889+0000","last_active":"2024-03-30T13:20:31.612532+0000","last_peered":"2024-03-30T13:20:31.612532+0000","last_clean":"2024-03-30T13:20:31.612532+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:20:31.612532+0000","last_undegraded":"2024-03-30T13:20:31.612532+0000","last_fullsized":"2024-03-30T13:20:31.612532+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:20:25.690651+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:55.652556+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1574,"reported_epoch":690,"state":"stale+active+clean","last_fresh":"2024-03-30T13:20:26.737553+0000","last_change":"2024-03-30T13:20:04.494258+0000","last_active":"2024-03-30T13:20:26.737553+0000","last_peered":"2024-03-30T13:20:26.737553+0000","last_clean":"2024-03-30T13:20:26.737553+0000","last_became_active":"2024-03-30T13:14:38.238193+0000","last_became_peered":"2024-03-30T13:14:38.238193+0000","last_unstale":"2024-03-30T13:20:31.610339+0000","last_undegraded":"2024-03-30T13:20:26.737553+0000","last_fullsized":"2024-03-30T13:20:26.737553+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:20:04.493986+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_clean_scrub_stamp":"2024-03-30T13:20:04.493986+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:09.894871+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":34184943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":275,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-03-30T13:20:31.612525+0000","last_change":"2024-03-30T13:20:14.423014+0000","last_active":"2024-03-30T13:20:31.612525+0000","last_peered":"2024-03-30T13:20:31.612525+0000","last_clean":"2024-03-30T13:20:31.612525+0000","last_became_active":"2024-03-30T13:20:14.421760+0000","last_became_peered":"2024-03-30T13:20:14.421760+0000","last_unstale":"2024-03-30T13:20:31.612525+0000","last_undegraded":"2024-03-30T13:20:31.612525+0000","last_fullsized":"2024-03-30T13:20:31.612525+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:19:35.780073+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:19:35.780073+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:41.822890+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":829,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-03-30T13:20:29.638697+0000","last_change":"2024-03-30T13:20:23.372796+0000","last_active":"2024-03-30T13:20:29.638697+0000","last_peered":"2024-03-30T13:20:29.638697+0000","last_clean":"2024-03-30T13:20:29.638697+0000","last_became_active":"2024-03-30T13:18:32.413904+0000","last_became_peered":"2024-03-30T13:18:32.413904+0000","last_unstale":"2024-03-30T13:20:29.638697+0000","last_undegraded":"2024-03-30T13:20:29.638697+0000","last_fullsized":"2024-03-30T13:20:29.638697+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:20:23.372460+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:20:23.372460+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:29.313113+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1219,"reported_epoch":694,"state":"peering","last_fresh":"2024-03-30T13:20:31.613763+0000","last_change":"2024-03-30T13:13:46.467647+0000","last_active":"2024-03-30T13:13:46.467647+0000","last_peered":"2024-03-30T13:13:43.044166+0000","last_clean":"2024-03-30T13:13:43.044166+0000","last_became_active":"2024-03-30T13:13:10.429866+0000","last_became_peered":"2024-03-30T13:13:10.429866+0000","last_unstale":"2024-03-30T13:20:31.613763+0000","last_undegraded":"2024-03-30T13:20:31.613763+0000","last_fullsized":"2024-03-30T13:20:31.613763+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:19:37.564154+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:19:37.564154+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2147483647,5,7],"acting":[2147483647,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1288,"reported_epoch":691,"state":"active+clean","last_fresh":"2024-03-30T13:20:27.855804+0000","last_change":"2024-03-30T13:20:23.631457+0000","last_active":"2024-03-30T13:20:27.855804+0000","last_peered":"2024-03-30T13:20:27.855804+0000","last_clean":"2024-03-30T13:20:27.855804+0000","last_became_active":"2024-03-30T13:20:14.427056+0000","last_became_peered":"2024-03-30T13:20:14.427056+0000","last_unstale":"2024-03-30T13:20:27.855804+0000","last_undegraded":"2024-03-30T13:20:27.855804+0000","last_fullsized":"2024-03-30T13:20:27.855804+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:20:23.631067+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:42.480355+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1593,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-03-30T13:20:29.642593+0000","last_change":"2024-03-30T13:20:14.433762+0000","last_active":"2024-03-30T13:20:29.642593+0000","last_peered":"2024-03-30T13:20:29.642593+0000","last_clean":"2024-03-30T13:20:29.642593+0000","last_became_active":"2024-03-30T13:20:14.431380+0000","last_became_peered":"2024-03-30T13:20:14.431380+0000","last_unstale":"2024-03-30T13:20:29.642593+0000","last_undegraded":"2024-03-30T13:20:29.642593+0000","last_fullsized":"2024-03-30T13:20:29.642593+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:19:19.392102+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:17:50.531859+0000","last_clean_scrub_stamp":"2024-03-30T13:19:19.392102+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:41.944162+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1621,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-03-30T13:20:29.642826+0000","last_change":"2024-03-30T13:19:05.199227+0000","last_active":"2024-03-30T13:20:29.642826+0000","last_peered":"2024-03-30T13:20:29.642826+0000","last_clean":"2024-03-30T13:20:29.642826+0000","last_became_active":"2024-03-30T13:18:32.419902+0000","last_became_peered":"2024-03-30T13:18:32.419902+0000","last_unstale":"2024-03-30T13:20:29.642826+0000","last_undegraded":"2024-03-30T13:20:29.642826+0000","last_fullsized":"2024-03-30T13:20:29.642826+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:19:05.198795+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:19:05.198795+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:32.312402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":432,"reported_epoch":693,"state":"peering","last_fresh":"2024-03-30T13:20:29.641630+0000","last_change":"2024-03-30T13:20:31.613446+0000","last_active":"2024-03-30T13:20:31.613446+0000","last_peered":"2024-03-30T13:20:29.641630+0000","last_clean":"2024-03-30T13:20:29.641630+0000","last_became_active":"2024-03-30T13:14:38.235735+0000","last_became_peered":"2024-03-30T13:14:38.235735+0000","last_unstale":"2024-03-30T13:20:29.641630+0000","last_undegraded":"2024-03-30T13:20:29.641630+0000","last_fullsized":"2024-03-30T13:20:29.641630+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:20:28.696431+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:20:28.696431+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,2147483647],"acting":[5,2,2147483647],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":213,"reported_epoch":693,"state":"peering","last_fresh":"2024-03-30T13:20:29.641987+0000","last_change":"2024-03-30T13:20:31.617069+0000","last_active":"2024-03-30T13:20:31.617069+0000","last_peered":"2024-03-30T13:20:29.641987+0000","last_clean":"2024-03-30T13:20:29.641987+0000","last_became_active":"2024-03-30T13:19:23.380857+0000","last_became_peered":"2024-03-30T13:19:23.380857+0000","last_unstale":"2024-03-30T13:20:29.641987+0000","last_undegraded":"2024-03-30T13:20:29.641987+0000","last_fullsized":"2024-03-30T13:20:29.641987+0000","mapping_epoch":694,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:19:48.759444+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:19:48.759444+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2147483647,2],"acting":[7,2147483647,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":431,"reported_epoch":694,"state":"active+clean","last_fresh":"2024-03-30T13:20:31.612183+0000","last_change":"2024-03-30T13:19:33.837451+0000","last_active":"2024-03-30T13:20:31.612183+0000","last_peered":"2024-03-30T13:20:31.612183+0000","last_clean":"2024-03-30T13:20:31.612183+0000","last_became_active":"2024-03-30T13:18:32.415413+0000","last_became_peered":"2024-03-30T13:18:32.415413+0000","last_unstale":"2024-03-30T13:20:31.612183+0000","last_undegraded":"2024-03-30T13:20:31.612183+0000","last_fullsized":"2024-03-30T13:20:31.612183+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:19:33.837082+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:19:33.837082+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:54.376722+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":782,"reported_epoch":693,"state":"active+clean","last_fresh":"2024-03-30T13:20:29.638910+0000","last_change":"2024-03-30T13:19:19.587739+0000","last_active":"2024-03-30T13:20:29.638910+0000","last_peered":"2024-03-30T13:20:29.638910+0000","last_clean":"2024-03-30T13:20:29.638910+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:20:29.638910+0000","last_undegraded":"2024-03-30T13:20:29.638910+0000","last_fullsized":"2024-03-30T13:20:29.638910+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:19:19.587471+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:19:19.587471+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:20:40.749837+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":252067987,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":629,"seq":2701534429210,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60132,"kb_used_data":23376,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94311708,"statfs":{"total":96636764160,"available":96575188992,"internally_reserved":0,"allocated":23937024,"data_stored":19290713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157513,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65776,"kb_used_data":28132,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94306064,"statfs":{"total":96636764160,"available":96569409536,"internally_reserved":0,"allocated":28807168,"data_stored":24154232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":677,"seq":2907692859397,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57440,"kb_used_data":19788,"kb_used_omap":13,"kb_used_meta":37618,"kb_avail":94314400,"statfs":{"total":96636764160,"available":96577945600,"internally_reserved":0,"allocated":20262912,"data_stored":15637004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14287,"internal_metadata":38520881},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509743,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49928,"kb_used_data":14580,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94321912,"statfs":{"total":96636764160,"available":96585637888,"internally_reserved":0,"allocated":14929920,"data_stored":10290410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375999022,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51052,"kb_used_data":13856,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94320788,"statfs":{"total":96636764160,"available":96584486912,"internally_reserved":0,"allocated":14188544,"data_stored":9549034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509744,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57112,"kb_used_data":21768,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94314728,"statfs":{"total":96636764160,"available":96578281472,"internally_reserved":0,"allocated":22290432,"data_stored":17644121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742416,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54752,"kb_used_data":18324,"kb_used_omap":15,"kb_used_meta":36400,"kb_avail":94317088,"statfs":{"total":96636764160,"available":96580698112,"internally_reserved":0,"allocated":18763776,"data_stored":14110840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":37274051},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:20:32.959 INFO:tasks.ceph.ceph_manager.ceph:PG 1.4 is not active or peered 2024-03-30T13:20:32.959 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.4', 'version': "22'6", 'reported_seq': 1219, 'reported_epoch': 694, 'state': 'peering', 'last_fresh': '2024-03-30T13:20:31.613763+0000', 'last_change': '2024-03-30T13:13:46.467647+0000', 'last_active': '2024-03-30T13:13:46.467647+0000', 'last_peered': '2024-03-30T13:13:43.044166+0000', 'last_clean': '2024-03-30T13:13:43.044166+0000', 'last_became_active': '2024-03-30T13:13:10.429866+0000', 'last_became_peered': '2024-03-30T13:13:10.429866+0000', 'last_unstale': '2024-03-30T13:20:31.613763+0000', 'last_undegraded': '2024-03-30T13:20:31.613763+0000', 'last_fullsized': '2024-03-30T13:20:31.613763+0000', 'mapping_epoch': 694, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 651, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "22'6", 'last_scrub_stamp': '2024-03-30T13:19:37.564154+0000', 'last_deep_scrub': "22'6", 'last_deep_scrub_stamp': '2024-03-30T13:16:45.572362+0000', 'last_clean_scrub_stamp': '2024-03-30T13:19:37.564154+0000', 'objects_scrubbed': 2, 'log_size': 6, 'log_dups_size': 0, 'ondisk_log_size': 6, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 7, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5075648, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 4, 'num_read_kb': 2, 'num_write': 8, 'num_write_kb': 2470, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24, 'num_bytes_recovered': 60907776, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2147483647, 5, 7], 'acting': [2147483647, 5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-30T13:20:32.959 INFO:tasks.ceph.ceph_manager.ceph:PG 1.8 is not active or peered 2024-03-30T13:20:32.959 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.8', 'version': "22'12", 'reported_seq': 432, 'reported_epoch': 693, 'state': 'peering', 'last_fresh': '2024-03-30T13:20:29.641630+0000', 'last_change': '2024-03-30T13:20:31.613446+0000', 'last_active': '2024-03-30T13:20:31.613446+0000', 'last_peered': '2024-03-30T13:20:29.641630+0000', 'last_clean': '2024-03-30T13:20:29.641630+0000', 'last_became_active': '2024-03-30T13:14:38.235735+0000', 'last_became_peered': '2024-03-30T13:14:38.235735+0000', 'last_unstale': '2024-03-30T13:20:29.641630+0000', 'last_undegraded': '2024-03-30T13:20:29.641630+0000', 'last_fullsized': '2024-03-30T13:20:29.641630+0000', 'mapping_epoch': 694, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 516, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "22'12", 'last_scrub_stamp': '2024-03-30T13:20:28.696431+0000', 'last_deep_scrub': "22'12", 'last_deep_scrub_stamp': '2024-03-30T13:07:53.685778+0000', 'last_clean_scrub_stamp': '2024-03-30T13:20:28.696431+0000', 'objects_scrubbed': 4, 'log_size': 12, 'log_dups_size': 0, 'ondisk_log_size': 12, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 7, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 9159683, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4, 'num_whiteouts': 0, 'num_read': 6, 'num_read_kb': 2, 'num_write': 16, 'num_write_kb': 3914, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2, 2147483647], 'acting': [5, 2, 2147483647], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-30T13:20:32.959 INFO:tasks.ceph.ceph_manager.ceph:PG 1.9 is not active or peered 2024-03-30T13:20:32.959 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.9', 'version': "22'3", 'reported_seq': 213, 'reported_epoch': 693, 'state': 'peering', 'last_fresh': '2024-03-30T13:20:29.641987+0000', 'last_change': '2024-03-30T13:20:31.617069+0000', 'last_active': '2024-03-30T13:20:31.617069+0000', 'last_peered': '2024-03-30T13:20:29.641987+0000', 'last_clean': '2024-03-30T13:20:29.641987+0000', 'last_became_active': '2024-03-30T13:19:23.380857+0000', 'last_became_peered': '2024-03-30T13:19:23.380857+0000', 'last_unstale': '2024-03-30T13:20:29.641987+0000', 'last_undegraded': '2024-03-30T13:20:29.641987+0000', 'last_fullsized': '2024-03-30T13:20:29.641987+0000', 'mapping_epoch': 694, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 651, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "22'3", 'last_scrub_stamp': '2024-03-30T13:19:48.759444+0000', 'last_deep_scrub': "22'3", 'last_deep_scrub_stamp': '2024-03-30T12:59:24.867416+0000', 'last_clean_scrub_stamp': '2024-03-30T13:19:48.759444+0000', 'objects_scrubbed': 1, 'log_size': 3, 'log_dups_size': 0, 'ondisk_log_size': 3, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 7, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2487403, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 3, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 0, 'num_write': 4, 'num_write_kb': 987, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2147483647, 2], 'acting': [7, 2147483647, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-30T13:20:32.960 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:20:57.960 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:20:57.961 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:20:58.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:58.089+0000 7fe3d4a14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:58.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:20:58.101+0000 7fe3d4a14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:58.277 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:20:58.278 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:20:58.295 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1452,"stamp":"2024-03-30T13:20:58.224421+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":267899983,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":409776,"kb_used_data":153456,"kb_used_omap":60,"kb_used_meta":256195,"kb_avail":660193104,"statfs":{"total":676457349120,"available":676037738496,"internally_reserved":0,"allocated":157138944,"data_stored":124474326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61447,"internal_metadata":262344697},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003417"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1625,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-03-30T13:20:33.635653+0000","last_change":"2024-03-30T13:20:14.433776+0000","last_active":"2024-03-30T13:20:33.635653+0000","last_peered":"2024-03-30T13:20:33.635653+0000","last_clean":"2024-03-30T13:20:33.635653+0000","last_became_active":"2024-03-30T13:20:14.431136+0000","last_became_peered":"2024-03-30T13:20:14.431136+0000","last_unstale":"2024-03-30T13:20:33.635653+0000","last_undegraded":"2024-03-30T13:20:33.635653+0000","last_fullsized":"2024-03-30T13:20:33.635653+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:02.484916+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:20:02.484916+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:25.338815+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1462,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-03-30T13:20:34.376183+0000","last_change":"2024-03-30T13:20:34.376183+0000","last_active":"2024-03-30T13:20:34.376183+0000","last_peered":"2024-03-30T13:20:34.376183+0000","last_clean":"2024-03-30T13:20:34.376183+0000","last_became_active":"2024-03-30T13:20:34.344033+0000","last_became_peered":"2024-03-30T13:20:34.344033+0000","last_unstale":"2024-03-30T13:20:34.376183+0000","last_undegraded":"2024-03-30T13:20:34.376183+0000","last_fullsized":"2024-03-30T13:20:34.376183+0000","mapping_epoch":695,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":696,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:19:43.319043+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:19:43.319043+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:03.416261+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":239,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-03-30T13:20:39.901176+0000","last_change":"2024-03-30T13:20:39.901251+0000","last_active":"2024-03-30T13:20:39.901176+0000","last_peered":"2024-03-30T13:20:39.901176+0000","last_clean":"2024-03-30T13:20:39.901176+0000","last_became_active":"2024-03-30T13:20:14.422234+0000","last_became_peered":"2024-03-30T13:20:14.422234+0000","last_unstale":"2024-03-30T13:20:39.901176+0000","last_undegraded":"2024-03-30T13:20:39.901176+0000","last_fullsized":"2024-03-30T13:20:39.901176+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:20:39.900992+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:20:39.900992+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:05.511322+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":227,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-03-30T13:20:33.628843+0000","last_change":"2024-03-30T13:20:10.775281+0000","last_active":"2024-03-30T13:20:33.628843+0000","last_peered":"2024-03-30T13:20:33.628843+0000","last_clean":"2024-03-30T13:20:33.628843+0000","last_became_active":"2024-03-30T13:19:23.949921+0000","last_became_peered":"2024-03-30T13:19:23.949921+0000","last_unstale":"2024-03-30T13:20:33.628843+0000","last_undegraded":"2024-03-30T13:20:33.628843+0000","last_fullsized":"2024-03-30T13:20:33.628843+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:10.775032+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:20:10.775032+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:18.932592+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":425,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-03-30T13:20:33.633572+0000","last_change":"2024-03-30T13:20:25.690889+0000","last_active":"2024-03-30T13:20:33.633572+0000","last_peered":"2024-03-30T13:20:33.633572+0000","last_clean":"2024-03-30T13:20:33.633572+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:20:33.633572+0000","last_undegraded":"2024-03-30T13:20:33.633572+0000","last_fullsized":"2024-03-30T13:20:33.633572+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:20:25.690651+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:55.652556+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":23,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-03-30T13:20:33.731087+0000","last_change":"2024-03-30T13:20:33.730152+0000","last_active":"2024-03-30T13:20:33.731087+0000","last_peered":"2024-03-30T13:20:33.731087+0000","last_clean":"2024-03-30T13:20:33.731087+0000","last_became_active":"2024-03-30T13:20:33.641593+0000","last_became_peered":"2024-03-30T13:20:33.641593+0000","last_unstale":"2024-03-30T13:20:33.731087+0000","last_undegraded":"2024-03-30T13:20:33.731087+0000","last_fullsized":"2024-03-30T13:20:33.731087+0000","mapping_epoch":695,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":696,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:20:04.493986+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_clean_scrub_stamp":"2024-03-30T13:20:04.493986+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:12.775397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11394981,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":287,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-03-30T13:20:45.976667+0000","last_change":"2024-03-30T13:20:45.976822+0000","last_active":"2024-03-30T13:20:45.976667+0000","last_peered":"2024-03-30T13:20:45.976667+0000","last_clean":"2024-03-30T13:20:45.976667+0000","last_became_active":"2024-03-30T13:20:14.421760+0000","last_became_peered":"2024-03-30T13:20:14.421760+0000","last_unstale":"2024-03-30T13:20:45.976667+0000","last_undegraded":"2024-03-30T13:20:45.976667+0000","last_fullsized":"2024-03-30T13:20:45.976667+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:20:45.976377+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:20:45.976377+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:56.308455+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":835,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-03-30T13:20:34.339748+0000","last_change":"2024-03-30T13:20:23.372796+0000","last_active":"2024-03-30T13:20:34.339748+0000","last_peered":"2024-03-30T13:20:34.339748+0000","last_clean":"2024-03-30T13:20:34.339748+0000","last_became_active":"2024-03-30T13:18:32.413904+0000","last_became_peered":"2024-03-30T13:18:32.413904+0000","last_unstale":"2024-03-30T13:20:34.339748+0000","last_undegraded":"2024-03-30T13:20:34.339748+0000","last_fullsized":"2024-03-30T13:20:34.339748+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:20:23.372460+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:20:23.372460+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:29.313113+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":25,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-03-30T13:20:56.407860+0000","last_change":"2024-03-30T13:20:56.407934+0000","last_active":"2024-03-30T13:20:56.407860+0000","last_peered":"2024-03-30T13:20:56.407860+0000","last_clean":"2024-03-30T13:20:56.407860+0000","last_became_active":"2024-03-30T13:20:33.641067+0000","last_became_peered":"2024-03-30T13:20:33.641067+0000","last_unstale":"2024-03-30T13:20:56.407860+0000","last_undegraded":"2024-03-30T13:20:56.407860+0000","last_fullsized":"2024-03-30T13:20:56.407860+0000","mapping_epoch":695,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":696,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:56.407671+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:20:56.407671+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:18.401481+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":65983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1298,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-03-30T13:20:33.632562+0000","last_change":"2024-03-30T13:20:23.631457+0000","last_active":"2024-03-30T13:20:33.632562+0000","last_peered":"2024-03-30T13:20:33.632562+0000","last_clean":"2024-03-30T13:20:33.632562+0000","last_became_active":"2024-03-30T13:20:14.427056+0000","last_became_peered":"2024-03-30T13:20:14.427056+0000","last_unstale":"2024-03-30T13:20:33.632562+0000","last_undegraded":"2024-03-30T13:20:33.632562+0000","last_fullsized":"2024-03-30T13:20:33.632562+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:20:23.631067+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:42.480355+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1608,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-03-30T13:20:42.398720+0000","last_change":"2024-03-30T13:20:42.398798+0000","last_active":"2024-03-30T13:20:42.398720+0000","last_peered":"2024-03-30T13:20:42.398720+0000","last_clean":"2024-03-30T13:20:42.398720+0000","last_became_active":"2024-03-30T13:20:14.431380+0000","last_became_peered":"2024-03-30T13:20:14.431380+0000","last_unstale":"2024-03-30T13:20:42.398720+0000","last_undegraded":"2024-03-30T13:20:42.398720+0000","last_fullsized":"2024-03-30T13:20:42.398720+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:20:42.398509+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:53.484365+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1659,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-03-30T13:20:35.456067+0000","last_change":"2024-03-30T13:20:35.456067+0000","last_active":"2024-03-30T13:20:35.456067+0000","last_peered":"2024-03-30T13:20:35.456067+0000","last_clean":"2024-03-30T13:20:35.456067+0000","last_became_active":"2024-03-30T13:20:33.644744+0000","last_became_peered":"2024-03-30T13:20:33.644744+0000","last_unstale":"2024-03-30T13:20:35.456067+0000","last_undegraded":"2024-03-30T13:20:35.456067+0000","last_fullsized":"2024-03-30T13:20:35.456067+0000","mapping_epoch":695,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":696,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:20:35.455810+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:20:35.455810+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:41.493713+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":452,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-03-30T13:20:33.730015+0000","last_change":"2024-03-30T13:20:33.730015+0000","last_active":"2024-03-30T13:20:33.730015+0000","last_peered":"2024-03-30T13:20:33.730015+0000","last_clean":"2024-03-30T13:20:33.730015+0000","last_became_active":"2024-03-30T13:20:33.641156+0000","last_became_peered":"2024-03-30T13:20:33.641156+0000","last_unstale":"2024-03-30T13:20:33.730015+0000","last_undegraded":"2024-03-30T13:20:33.730015+0000","last_fullsized":"2024-03-30T13:20:33.730015+0000","mapping_epoch":695,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":696,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:20:28.696431+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:20:28.696431+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:43.529836+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":227,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-03-30T13:20:34.109284+0000","last_change":"2024-03-30T13:20:34.109284+0000","last_active":"2024-03-30T13:20:34.109284+0000","last_peered":"2024-03-30T13:20:34.109284+0000","last_clean":"2024-03-30T13:20:34.109284+0000","last_became_active":"2024-03-30T13:20:33.639672+0000","last_became_peered":"2024-03-30T13:20:33.639672+0000","last_unstale":"2024-03-30T13:20:34.109284+0000","last_undegraded":"2024-03-30T13:20:34.109284+0000","last_fullsized":"2024-03-30T13:20:34.109284+0000","mapping_epoch":695,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":696,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:19:48.759444+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:19:48.759444+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:18.221611+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":443,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-03-30T13:20:54.701752+0000","last_change":"2024-03-30T13:20:54.701831+0000","last_active":"2024-03-30T13:20:54.701752+0000","last_peered":"2024-03-30T13:20:54.701752+0000","last_clean":"2024-03-30T13:20:54.701752+0000","last_became_active":"2024-03-30T13:18:32.415413+0000","last_became_peered":"2024-03-30T13:18:32.415413+0000","last_unstale":"2024-03-30T13:20:54.701752+0000","last_undegraded":"2024-03-30T13:20:54.701752+0000","last_fullsized":"2024-03-30T13:20:54.701752+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:20:54.701569+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:20:54.701569+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:18.695855+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":796,"reported_epoch":696,"state":"active+clean","last_fresh":"2024-03-30T13:20:41.484144+0000","last_change":"2024-03-30T13:20:41.484219+0000","last_active":"2024-03-30T13:20:41.484144+0000","last_peered":"2024-03-30T13:20:41.484144+0000","last_clean":"2024-03-30T13:20:41.484144+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:20:41.484144+0000","last_undegraded":"2024-03-30T13:20:41.484144+0000","last_fullsized":"2024-03-30T13:20:41.484144+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:41.483960+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:20:41.483960+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:59.137796+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":267899983,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":110415872,"data_stored":110415872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":629,"seq":2701534429215,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65272,"kb_used_data":28528,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94306568,"statfs":{"total":96636764160,"available":96569925632,"internally_reserved":0,"allocated":29212672,"data_stored":24546298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157518,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65796,"kb_used_data":28156,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94306044,"statfs":{"total":96636764160,"available":96569389056,"internally_reserved":0,"allocated":28831744,"data_stored":24165370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":677,"seq":2907692859402,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57500,"kb_used_data":19860,"kb_used_omap":13,"kb_used_meta":37618,"kb_avail":94314340,"statfs":{"total":96636764160,"available":96577884160,"internally_reserved":0,"allocated":20336640,"data_stored":15670266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14287,"internal_metadata":38520881},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509748,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49972,"kb_used_data":14628,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94321868,"statfs":{"total":96636764160,"available":96585592832,"internally_reserved":0,"allocated":14979072,"data_stored":10312698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375999027,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51756,"kb_used_data":14564,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94320084,"statfs":{"total":96636764160,"available":96583766016,"internally_reserved":0,"allocated":14913536,"data_stored":10247162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509749,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64708,"kb_used_data":29372,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94307132,"statfs":{"total":96636764160,"available":96570503168,"internally_reserved":0,"allocated":30076928,"data_stored":25410554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742422,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54772,"kb_used_data":18348,"kb_used_omap":15,"kb_used_meta":36400,"kb_avail":94317068,"statfs":{"total":96636764160,"available":96580677632,"internally_reserved":0,"allocated":18788352,"data_stored":14121978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":37274051},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21573632,"data_stored":21573632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6410240,"data_stored":6410240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20709376,"data_stored":20709376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:20:58.297 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:20:58.297 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:20:58.297 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-03-30T13:20:58.297 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-30T13:20:58.297 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-30T13:20:58.299 INFO:tasks.ceph.osd.3:Started 2024-03-30T13:20:58.300 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-30T13:20:58.357 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T13:20:58.357+0000 7f289cb417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:58.363 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T13:20:58.361+0000 7f289cb417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:58.363 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T13:20:58.361+0000 7f289cb417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:20:58.409 INFO:teuthology.orchestra.run.smithi067.stderr:no valid command found; 10 closest matches: 2024-03-30T13:20:58.410 INFO:teuthology.orchestra.run.smithi067.stderr:0 2024-03-30T13:20:58.410 INFO:teuthology.orchestra.run.smithi067.stderr:1 2024-03-30T13:20:58.410 INFO:teuthology.orchestra.run.smithi067.stderr:2 2024-03-30T13:20:58.410 INFO:teuthology.orchestra.run.smithi067.stderr:abort 2024-03-30T13:20:58.410 INFO:teuthology.orchestra.run.smithi067.stderr:assert 2024-03-30T13:20:58.410 INFO:teuthology.orchestra.run.smithi067.stderr:config diff 2024-03-30T13:20:58.410 INFO:teuthology.orchestra.run.smithi067.stderr:config diff get 2024-03-30T13:20:58.410 INFO:teuthology.orchestra.run.smithi067.stderr:config get 2024-03-30T13:20:58.410 INFO:teuthology.orchestra.run.smithi067.stderr:config help [] 2024-03-30T13:20:58.410 INFO:teuthology.orchestra.run.smithi067.stderr:config set ... 2024-03-30T13:20:58.410 INFO:teuthology.orchestra.run.smithi067.stderr:admin_socket: invalid command 2024-03-30T13:20:58.412 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:20:58.412 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-03-30T13:20:58.922 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T13:20:58.921+0000 7f289cb417c0 -1 Falling back to public interface 2024-03-30T13:21:02.696 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T13:21:02.697+0000 7f289cb417c0 -1 osd.3 693 log_to_monitors true 2024-03-30T13:21:03.413 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-30T13:21:03.560 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T13:21:03.561+0000 7f2894916640 -1 osd.3 693 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:21:03.577 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-30T13:21:03.577 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-30T13:21:03.577 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-03-30T13:21:03.578 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-30T13:21:03.592 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:21:03.592 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-03-30T13:21:03.592 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 3 2024-03-30T13:21:03.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:03.709+0000 7fceb59ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:03.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:03.725+0000 7fceb59ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:03.907 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:21:03.905+0000 7fbb38f08640 -1 mon.a@0(leader).osd e698 definitely_dead 0 2024-03-30T13:21:04.548 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.3. 2024-03-30T13:21:04.581 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-03-30T13:21:04.581 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:21:04.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:04.693+0000 7f452fb9f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:04.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:04.709+0000 7f452fb9f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:05.561 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:21:05.588 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:21:05.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:05.705+0000 7f85d1741640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:05.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:05.721+0000 7f85d1741640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:06.585 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:21:06.607 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:21:06.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:06.721+0000 7fad453a1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:06.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:06.737+0000 7fad453a1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:07.581 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:21:07.607 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:21:07.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:07.729+0000 7f32adb86640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:07.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:07.745+0000 7f32adb86640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:08.589 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:21:08.613 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:21:08.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:08.729+0000 7f1f25cdc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:08.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:08.749+0000 7f1f25cdc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:09.592 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:21:09.618 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:21:09.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:09.737+0000 7f3b72b91640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:09.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:09.757+0000 7f3b72b91640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:10.603 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:21:10.631 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:21:10.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:10.753+0000 7f1d52c65640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:10.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:10.769+0000 7f1d52c65640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:11.620 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:21:11.644 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:21:11.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:11.765+0000 7fcacd222640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:11.781+0000 7fcacd222640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:12.631 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:21:12.660 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:21:12.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:12.781+0000 7f8452057640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:12.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:12.797+0000 7f8452057640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:13.636 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:21:13.666 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:21:13.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:13.785+0000 7f5935641640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:13.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:13.805+0000 7f5935641640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:14.661 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:21:14.685 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:21:14.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:14.801+0000 7efceb25d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:14.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:14.817+0000 7efceb25d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:15.680 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:21:15.708 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:21:15.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:15.829+0000 7f7b76fe7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:15.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:15.845+0000 7f7b76fe7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:16.681 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:21:16.706 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:21:16.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:16.825+0000 7f80d2a2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:16.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:16.841+0000 7f80d2a2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:17.690 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:21:17.714 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:21:17.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:17.833+0000 7f2f34d70640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:17.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:17.853+0000 7f2f34d70640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:18.702 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:21:18.731 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:21:18.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:18.849+0000 7fd6a35f4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:18.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:18.865+0000 7fd6a35f4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:19.705 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:21:19.732 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:21:19.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:19.849+0000 7f2c77f2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:19.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:19.869+0000 7f2c77f2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:20.722 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:21:20.747 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:21:20.748 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:21:20.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:20.865+0000 7f38259a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:20.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:20.877+0000 7f38259a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:21.052 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:21:21.052 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:21:21.070 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1482,"stamp":"2024-03-30T13:21:20.726333+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":251429354,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":455056,"kb_used_data":162380,"kb_used_omap":65,"kb_used_meta":292478,"kb_avail":754519664,"statfs":{"total":773094113280,"available":772628135936,"internally_reserved":0,"allocated":166277120,"data_stored":128130442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67493,"internal_metadata":299497563},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.040627"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1650,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-03-30T13:21:14.657545+0000","last_change":"2024-03-30T13:20:14.433776+0000","last_active":"2024-03-30T13:21:14.657545+0000","last_peered":"2024-03-30T13:21:14.657545+0000","last_clean":"2024-03-30T13:21:14.657545+0000","last_became_active":"2024-03-30T13:20:14.431136+0000","last_became_peered":"2024-03-30T13:20:14.431136+0000","last_unstale":"2024-03-30T13:21:14.657545+0000","last_undegraded":"2024-03-30T13:21:14.657545+0000","last_fullsized":"2024-03-30T13:21:14.657545+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:02.484916+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:20:02.484916+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:25.338815+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1508,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-03-30T13:21:17.690118+0000","last_change":"2024-03-30T13:21:17.333015+0000","last_active":"2024-03-30T13:21:17.690118+0000","last_peered":"2024-03-30T13:21:17.690118+0000","last_clean":"2024-03-30T13:21:17.690118+0000","last_became_active":"2024-03-30T13:21:05.571151+0000","last_became_peered":"2024-03-30T13:21:05.571151+0000","last_unstale":"2024-03-30T13:21:17.690118+0000","last_undegraded":"2024-03-30T13:21:17.690118+0000","last_fullsized":"2024-03-30T13:21:17.690118+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:17.332777+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:21:17.332777+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:44.049961+0000","scrub_duration":695,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":268,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-03-30T13:21:17.328486+0000","last_change":"2024-03-30T13:20:39.901251+0000","last_active":"2024-03-30T13:21:17.328486+0000","last_peered":"2024-03-30T13:21:17.328486+0000","last_clean":"2024-03-30T13:21:17.328486+0000","last_became_active":"2024-03-30T13:20:14.422234+0000","last_became_peered":"2024-03-30T13:20:14.422234+0000","last_unstale":"2024-03-30T13:21:17.328486+0000","last_undegraded":"2024-03-30T13:21:17.328486+0000","last_fullsized":"2024-03-30T13:21:17.328486+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:20:39.900992+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:20:39.900992+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:05.511322+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":256,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-03-30T13:21:17.328389+0000","last_change":"2024-03-30T13:20:10.775281+0000","last_active":"2024-03-30T13:21:17.328389+0000","last_peered":"2024-03-30T13:21:17.328389+0000","last_clean":"2024-03-30T13:21:17.328389+0000","last_became_active":"2024-03-30T13:19:23.949921+0000","last_became_peered":"2024-03-30T13:19:23.949921+0000","last_unstale":"2024-03-30T13:21:17.328389+0000","last_undegraded":"2024-03-30T13:21:17.328389+0000","last_fullsized":"2024-03-30T13:21:17.328389+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:10.775032+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:20:10.775032+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:18.932592+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":451,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-03-30T13:21:16.599612+0000","last_change":"2024-03-30T13:20:25.690889+0000","last_active":"2024-03-30T13:21:16.599612+0000","last_peered":"2024-03-30T13:21:16.599612+0000","last_clean":"2024-03-30T13:21:16.599612+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:21:16.599612+0000","last_undegraded":"2024-03-30T13:21:16.599612+0000","last_fullsized":"2024-03-30T13:21:16.599612+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:20:25.690651+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:55.652556+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1579,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-03-30T13:21:18.710485+0000","last_change":"2024-03-30T13:21:05.574895+0000","last_active":"2024-03-30T13:21:18.710485+0000","last_peered":"2024-03-30T13:21:18.710485+0000","last_clean":"2024-03-30T13:21:18.710485+0000","last_became_active":"2024-03-30T13:21:05.574411+0000","last_became_peered":"2024-03-30T13:21:05.574411+0000","last_unstale":"2024-03-30T13:21:18.710485+0000","last_undegraded":"2024-03-30T13:21:18.710485+0000","last_fullsized":"2024-03-30T13:21:18.710485+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:20:04.493986+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_clean_scrub_stamp":"2024-03-30T13:20:04.493986+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:20.538548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":316,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-03-30T13:21:17.328431+0000","last_change":"2024-03-30T13:20:45.976822+0000","last_active":"2024-03-30T13:21:17.328431+0000","last_peered":"2024-03-30T13:21:17.328431+0000","last_clean":"2024-03-30T13:21:17.328431+0000","last_became_active":"2024-03-30T13:20:14.421760+0000","last_became_peered":"2024-03-30T13:20:14.421760+0000","last_unstale":"2024-03-30T13:21:17.328431+0000","last_undegraded":"2024-03-30T13:21:17.328431+0000","last_fullsized":"2024-03-30T13:21:17.328431+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:20:45.976377+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:20:45.976377+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:56.308455+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":872,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-03-30T13:21:19.714145+0000","last_change":"2024-03-30T13:20:23.372796+0000","last_active":"2024-03-30T13:21:19.714145+0000","last_peered":"2024-03-30T13:21:19.714145+0000","last_clean":"2024-03-30T13:21:19.714145+0000","last_became_active":"2024-03-30T13:18:32.413904+0000","last_became_peered":"2024-03-30T13:18:32.413904+0000","last_unstale":"2024-03-30T13:21:19.714145+0000","last_undegraded":"2024-03-30T13:21:19.714145+0000","last_fullsized":"2024-03-30T13:21:19.714145+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:20:23.372460+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:20:23.372460+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:29.313113+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1718,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-03-30T13:21:18.710404+0000","last_change":"2024-03-30T13:21:05.574623+0000","last_active":"2024-03-30T13:21:18.710404+0000","last_peered":"2024-03-30T13:21:18.710404+0000","last_clean":"2024-03-30T13:21:18.710404+0000","last_became_active":"2024-03-30T13:21:05.574131+0000","last_became_peered":"2024-03-30T13:21:05.574131+0000","last_unstale":"2024-03-30T13:21:18.710404+0000","last_undegraded":"2024-03-30T13:21:18.710404+0000","last_fullsized":"2024-03-30T13:21:18.710404+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:56.407671+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:20:56.407671+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:11.138773+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1330,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-03-30T13:21:17.690161+0000","last_change":"2024-03-30T13:20:23.631457+0000","last_active":"2024-03-30T13:21:17.690161+0000","last_peered":"2024-03-30T13:21:17.690161+0000","last_clean":"2024-03-30T13:21:17.690161+0000","last_became_active":"2024-03-30T13:20:14.427056+0000","last_became_peered":"2024-03-30T13:20:14.427056+0000","last_unstale":"2024-03-30T13:21:17.690161+0000","last_undegraded":"2024-03-30T13:21:17.690161+0000","last_fullsized":"2024-03-30T13:21:17.690161+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:20:23.631067+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:42.480355+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1633,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-03-30T13:21:14.657414+0000","last_change":"2024-03-30T13:20:42.398798+0000","last_active":"2024-03-30T13:21:14.657414+0000","last_peered":"2024-03-30T13:21:14.657414+0000","last_clean":"2024-03-30T13:21:14.657414+0000","last_became_active":"2024-03-30T13:20:14.431380+0000","last_became_peered":"2024-03-30T13:20:14.431380+0000","last_unstale":"2024-03-30T13:21:14.657414+0000","last_undegraded":"2024-03-30T13:21:14.657414+0000","last_fullsized":"2024-03-30T13:21:14.657414+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:20:42.398509+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:53.484365+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1690,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-03-30T13:21:14.657498+0000","last_change":"2024-03-30T13:21:05.572623+0000","last_active":"2024-03-30T13:21:14.657498+0000","last_peered":"2024-03-30T13:21:14.657498+0000","last_clean":"2024-03-30T13:21:14.657498+0000","last_became_active":"2024-03-30T13:21:05.570917+0000","last_became_peered":"2024-03-30T13:21:05.570917+0000","last_unstale":"2024-03-30T13:21:14.657498+0000","last_undegraded":"2024-03-30T13:21:14.657498+0000","last_fullsized":"2024-03-30T13:21:14.657498+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:20:35.455810+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:20:35.455810+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:00.400375+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":485,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-03-30T13:21:16.599176+0000","last_change":"2024-03-30T13:21:05.573717+0000","last_active":"2024-03-30T13:21:16.599176+0000","last_peered":"2024-03-30T13:21:16.599176+0000","last_clean":"2024-03-30T13:21:16.599176+0000","last_became_active":"2024-03-30T13:21:05.572577+0000","last_became_peered":"2024-03-30T13:21:05.572577+0000","last_unstale":"2024-03-30T13:21:16.599176+0000","last_undegraded":"2024-03-30T13:21:16.599176+0000","last_fullsized":"2024-03-30T13:21:16.599176+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:20:28.696431+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:20:28.696431+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:31.562367+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":270,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-03-30T13:21:17.328481+0000","last_change":"2024-03-30T13:21:06.773969+0000","last_active":"2024-03-30T13:21:17.328481+0000","last_peered":"2024-03-30T13:21:17.328481+0000","last_clean":"2024-03-30T13:21:17.328481+0000","last_became_active":"2024-03-30T13:21:05.570131+0000","last_became_peered":"2024-03-30T13:21:05.570131+0000","last_unstale":"2024-03-30T13:21:17.328481+0000","last_undegraded":"2024-03-30T13:21:17.328481+0000","last_fullsized":"2024-03-30T13:21:17.328481+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:06.773612+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:21:06.773612+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:19.252741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":470,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-03-30T13:21:16.599658+0000","last_change":"2024-03-30T13:20:54.701831+0000","last_active":"2024-03-30T13:21:16.599658+0000","last_peered":"2024-03-30T13:21:16.599658+0000","last_clean":"2024-03-30T13:21:16.599658+0000","last_became_active":"2024-03-30T13:18:32.415413+0000","last_became_peered":"2024-03-30T13:18:32.415413+0000","last_unstale":"2024-03-30T13:21:16.599658+0000","last_undegraded":"2024-03-30T13:21:16.599658+0000","last_fullsized":"2024-03-30T13:21:16.599658+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:20:54.701569+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:20:54.701569+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:18.695855+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":833,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-03-30T13:21:19.714328+0000","last_change":"2024-03-30T13:20:41.484219+0000","last_active":"2024-03-30T13:21:19.714328+0000","last_peered":"2024-03-30T13:21:19.714328+0000","last_clean":"2024-03-30T13:21:19.714328+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:21:19.714328+0000","last_undegraded":"2024-03-30T13:21:19.714328+0000","last_fullsized":"2024-03-30T13:21:19.714328+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:41.483960+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:20:41.483960+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:59.137796+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":251429354,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":629,"seq":2701534429219,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60320,"kb_used_data":23568,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94311520,"statfs":{"total":96636764160,"available":96574996480,"internally_reserved":0,"allocated":24133632,"data_stored":19380562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157522,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65960,"kb_used_data":28312,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94305880,"statfs":{"total":96636764160,"available":96569221120,"internally_reserved":0,"allocated":28991488,"data_stored":24238418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":677,"seq":2907692859407,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57692,"kb_used_data":20040,"kb_used_omap":13,"kb_used_meta":37618,"kb_avail":94314148,"statfs":{"total":96636764160,"available":96577687552,"internally_reserved":0,"allocated":20520960,"data_stored":15754338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14287,"internal_metadata":38520881},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509753,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50176,"kb_used_data":14832,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94321664,"statfs":{"total":96636764160,"available":96585383936,"internally_reserved":0,"allocated":15187968,"data_stored":10407744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375999032,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51312,"kb_used_data":14108,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94320528,"statfs":{"total":96636764160,"available":96584220672,"internally_reserved":0,"allocated":14446592,"data_stored":9666368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509754,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57356,"kb_used_data":22008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94314484,"statfs":{"total":96636764160,"available":96578031616,"internally_reserved":0,"allocated":22536192,"data_stored":17755968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742426,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59104,"kb_used_data":18516,"kb_used_omap":15,"kb_used_meta":40560,"kb_avail":94312736,"statfs":{"total":96636764160,"available":96576241664,"internally_reserved":0,"allocated":18960384,"data_stored":14200563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":41533891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":699,"seq":3002182139909,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53136,"kb_used_data":20996,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94318704,"statfs":{"total":96636764160,"available":96582352896,"internally_reserved":0,"allocated":21499904,"data_stored":16726481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:21:21.071 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:21:21.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:21.181+0000 7fd317f88640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:21.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:21.193+0000 7fd317f88640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:21.357 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:21:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:21:21.374 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1482,"stamp":"2024-03-30T13:21:20.726333+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":251429354,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":455056,"kb_used_data":162380,"kb_used_omap":65,"kb_used_meta":292478,"kb_avail":754519664,"statfs":{"total":773094113280,"available":772628135936,"internally_reserved":0,"allocated":166277120,"data_stored":128130442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67493,"internal_metadata":299497563},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.040627"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1650,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-03-30T13:21:14.657545+0000","last_change":"2024-03-30T13:20:14.433776+0000","last_active":"2024-03-30T13:21:14.657545+0000","last_peered":"2024-03-30T13:21:14.657545+0000","last_clean":"2024-03-30T13:21:14.657545+0000","last_became_active":"2024-03-30T13:20:14.431136+0000","last_became_peered":"2024-03-30T13:20:14.431136+0000","last_unstale":"2024-03-30T13:21:14.657545+0000","last_undegraded":"2024-03-30T13:21:14.657545+0000","last_fullsized":"2024-03-30T13:21:14.657545+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:02.484916+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:20:02.484916+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:25.338815+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1508,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-03-30T13:21:17.690118+0000","last_change":"2024-03-30T13:21:17.333015+0000","last_active":"2024-03-30T13:21:17.690118+0000","last_peered":"2024-03-30T13:21:17.690118+0000","last_clean":"2024-03-30T13:21:17.690118+0000","last_became_active":"2024-03-30T13:21:05.571151+0000","last_became_peered":"2024-03-30T13:21:05.571151+0000","last_unstale":"2024-03-30T13:21:17.690118+0000","last_undegraded":"2024-03-30T13:21:17.690118+0000","last_fullsized":"2024-03-30T13:21:17.690118+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:17.332777+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:21:17.332777+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:44.049961+0000","scrub_duration":695,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":268,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-03-30T13:21:17.328486+0000","last_change":"2024-03-30T13:20:39.901251+0000","last_active":"2024-03-30T13:21:17.328486+0000","last_peered":"2024-03-30T13:21:17.328486+0000","last_clean":"2024-03-30T13:21:17.328486+0000","last_became_active":"2024-03-30T13:20:14.422234+0000","last_became_peered":"2024-03-30T13:20:14.422234+0000","last_unstale":"2024-03-30T13:21:17.328486+0000","last_undegraded":"2024-03-30T13:21:17.328486+0000","last_fullsized":"2024-03-30T13:21:17.328486+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:20:39.900992+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:20:39.900992+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:05.511322+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":256,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-03-30T13:21:17.328389+0000","last_change":"2024-03-30T13:20:10.775281+0000","last_active":"2024-03-30T13:21:17.328389+0000","last_peered":"2024-03-30T13:21:17.328389+0000","last_clean":"2024-03-30T13:21:17.328389+0000","last_became_active":"2024-03-30T13:19:23.949921+0000","last_became_peered":"2024-03-30T13:19:23.949921+0000","last_unstale":"2024-03-30T13:21:17.328389+0000","last_undegraded":"2024-03-30T13:21:17.328389+0000","last_fullsized":"2024-03-30T13:21:17.328389+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:10.775032+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:20:10.775032+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:18.932592+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":451,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-03-30T13:21:16.599612+0000","last_change":"2024-03-30T13:20:25.690889+0000","last_active":"2024-03-30T13:21:16.599612+0000","last_peered":"2024-03-30T13:21:16.599612+0000","last_clean":"2024-03-30T13:21:16.599612+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:21:16.599612+0000","last_undegraded":"2024-03-30T13:21:16.599612+0000","last_fullsized":"2024-03-30T13:21:16.599612+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:20:25.690651+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:55.652556+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1579,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-03-30T13:21:18.710485+0000","last_change":"2024-03-30T13:21:05.574895+0000","last_active":"2024-03-30T13:21:18.710485+0000","last_peered":"2024-03-30T13:21:18.710485+0000","last_clean":"2024-03-30T13:21:18.710485+0000","last_became_active":"2024-03-30T13:21:05.574411+0000","last_became_peered":"2024-03-30T13:21:05.574411+0000","last_unstale":"2024-03-30T13:21:18.710485+0000","last_undegraded":"2024-03-30T13:21:18.710485+0000","last_fullsized":"2024-03-30T13:21:18.710485+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:20:04.493986+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_clean_scrub_stamp":"2024-03-30T13:20:04.493986+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:20.538548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":316,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-03-30T13:21:17.328431+0000","last_change":"2024-03-30T13:20:45.976822+0000","last_active":"2024-03-30T13:21:17.328431+0000","last_peered":"2024-03-30T13:21:17.328431+0000","last_clean":"2024-03-30T13:21:17.328431+0000","last_became_active":"2024-03-30T13:20:14.421760+0000","last_became_peered":"2024-03-30T13:20:14.421760+0000","last_unstale":"2024-03-30T13:21:17.328431+0000","last_undegraded":"2024-03-30T13:21:17.328431+0000","last_fullsized":"2024-03-30T13:21:17.328431+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:20:45.976377+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:20:45.976377+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:56.308455+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":872,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-03-30T13:21:19.714145+0000","last_change":"2024-03-30T13:20:23.372796+0000","last_active":"2024-03-30T13:21:19.714145+0000","last_peered":"2024-03-30T13:21:19.714145+0000","last_clean":"2024-03-30T13:21:19.714145+0000","last_became_active":"2024-03-30T13:18:32.413904+0000","last_became_peered":"2024-03-30T13:18:32.413904+0000","last_unstale":"2024-03-30T13:21:19.714145+0000","last_undegraded":"2024-03-30T13:21:19.714145+0000","last_fullsized":"2024-03-30T13:21:19.714145+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:20:23.372460+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:20:23.372460+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:29.313113+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1718,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-03-30T13:21:18.710404+0000","last_change":"2024-03-30T13:21:05.574623+0000","last_active":"2024-03-30T13:21:18.710404+0000","last_peered":"2024-03-30T13:21:18.710404+0000","last_clean":"2024-03-30T13:21:18.710404+0000","last_became_active":"2024-03-30T13:21:05.574131+0000","last_became_peered":"2024-03-30T13:21:05.574131+0000","last_unstale":"2024-03-30T13:21:18.710404+0000","last_undegraded":"2024-03-30T13:21:18.710404+0000","last_fullsized":"2024-03-30T13:21:18.710404+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:56.407671+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:20:56.407671+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:11.138773+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1330,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-03-30T13:21:17.690161+0000","last_change":"2024-03-30T13:20:23.631457+0000","last_active":"2024-03-30T13:21:17.690161+0000","last_peered":"2024-03-30T13:21:17.690161+0000","last_clean":"2024-03-30T13:21:17.690161+0000","last_became_active":"2024-03-30T13:20:14.427056+0000","last_became_peered":"2024-03-30T13:20:14.427056+0000","last_unstale":"2024-03-30T13:21:17.690161+0000","last_undegraded":"2024-03-30T13:21:17.690161+0000","last_fullsized":"2024-03-30T13:21:17.690161+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:20:23.631067+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:42.480355+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1633,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-03-30T13:21:14.657414+0000","last_change":"2024-03-30T13:20:42.398798+0000","last_active":"2024-03-30T13:21:14.657414+0000","last_peered":"2024-03-30T13:21:14.657414+0000","last_clean":"2024-03-30T13:21:14.657414+0000","last_became_active":"2024-03-30T13:20:14.431380+0000","last_became_peered":"2024-03-30T13:20:14.431380+0000","last_unstale":"2024-03-30T13:21:14.657414+0000","last_undegraded":"2024-03-30T13:21:14.657414+0000","last_fullsized":"2024-03-30T13:21:14.657414+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:20:42.398509+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:53.484365+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1690,"reported_epoch":709,"state":"active+clean","last_fresh":"2024-03-30T13:21:14.657498+0000","last_change":"2024-03-30T13:21:05.572623+0000","last_active":"2024-03-30T13:21:14.657498+0000","last_peered":"2024-03-30T13:21:14.657498+0000","last_clean":"2024-03-30T13:21:14.657498+0000","last_became_active":"2024-03-30T13:21:05.570917+0000","last_became_peered":"2024-03-30T13:21:05.570917+0000","last_unstale":"2024-03-30T13:21:14.657498+0000","last_undegraded":"2024-03-30T13:21:14.657498+0000","last_fullsized":"2024-03-30T13:21:14.657498+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:20:35.455810+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:20:35.455810+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:00.400375+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":485,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-03-30T13:21:16.599176+0000","last_change":"2024-03-30T13:21:05.573717+0000","last_active":"2024-03-30T13:21:16.599176+0000","last_peered":"2024-03-30T13:21:16.599176+0000","last_clean":"2024-03-30T13:21:16.599176+0000","last_became_active":"2024-03-30T13:21:05.572577+0000","last_became_peered":"2024-03-30T13:21:05.572577+0000","last_unstale":"2024-03-30T13:21:16.599176+0000","last_undegraded":"2024-03-30T13:21:16.599176+0000","last_fullsized":"2024-03-30T13:21:16.599176+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:20:28.696431+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:20:28.696431+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:31.562367+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":270,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-03-30T13:21:17.328481+0000","last_change":"2024-03-30T13:21:06.773969+0000","last_active":"2024-03-30T13:21:17.328481+0000","last_peered":"2024-03-30T13:21:17.328481+0000","last_clean":"2024-03-30T13:21:17.328481+0000","last_became_active":"2024-03-30T13:21:05.570131+0000","last_became_peered":"2024-03-30T13:21:05.570131+0000","last_unstale":"2024-03-30T13:21:17.328481+0000","last_undegraded":"2024-03-30T13:21:17.328481+0000","last_fullsized":"2024-03-30T13:21:17.328481+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:06.773612+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:21:06.773612+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:19.252741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":470,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-03-30T13:21:16.599658+0000","last_change":"2024-03-30T13:20:54.701831+0000","last_active":"2024-03-30T13:21:16.599658+0000","last_peered":"2024-03-30T13:21:16.599658+0000","last_clean":"2024-03-30T13:21:16.599658+0000","last_became_active":"2024-03-30T13:18:32.415413+0000","last_became_peered":"2024-03-30T13:18:32.415413+0000","last_unstale":"2024-03-30T13:21:16.599658+0000","last_undegraded":"2024-03-30T13:21:16.599658+0000","last_fullsized":"2024-03-30T13:21:16.599658+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:20:54.701569+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:20:54.701569+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:18.695855+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":833,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-03-30T13:21:19.714328+0000","last_change":"2024-03-30T13:20:41.484219+0000","last_active":"2024-03-30T13:21:19.714328+0000","last_peered":"2024-03-30T13:21:19.714328+0000","last_clean":"2024-03-30T13:21:19.714328+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:21:19.714328+0000","last_undegraded":"2024-03-30T13:21:19.714328+0000","last_fullsized":"2024-03-30T13:21:19.714328+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:41.483960+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:20:41.483960+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:59.137796+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":251429354,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":629,"seq":2701534429219,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60320,"kb_used_data":23568,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94311520,"statfs":{"total":96636764160,"available":96574996480,"internally_reserved":0,"allocated":24133632,"data_stored":19380562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157522,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65960,"kb_used_data":28312,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94305880,"statfs":{"total":96636764160,"available":96569221120,"internally_reserved":0,"allocated":28991488,"data_stored":24238418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":677,"seq":2907692859407,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57692,"kb_used_data":20040,"kb_used_omap":13,"kb_used_meta":37618,"kb_avail":94314148,"statfs":{"total":96636764160,"available":96577687552,"internally_reserved":0,"allocated":20520960,"data_stored":15754338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14287,"internal_metadata":38520881},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509753,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50176,"kb_used_data":14832,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94321664,"statfs":{"total":96636764160,"available":96585383936,"internally_reserved":0,"allocated":15187968,"data_stored":10407744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375999032,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51312,"kb_used_data":14108,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94320528,"statfs":{"total":96636764160,"available":96584220672,"internally_reserved":0,"allocated":14446592,"data_stored":9666368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509754,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57356,"kb_used_data":22008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94314484,"statfs":{"total":96636764160,"available":96578031616,"internally_reserved":0,"allocated":22536192,"data_stored":17755968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742426,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59104,"kb_used_data":18516,"kb_used_omap":15,"kb_used_meta":40560,"kb_avail":94312736,"statfs":{"total":96636764160,"available":96576241664,"internally_reserved":0,"allocated":18960384,"data_stored":14200563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":41533891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":699,"seq":3002182139909,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53136,"kb_used_data":20996,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94318704,"statfs":{"total":96636764160,"available":96582352896,"internally_reserved":0,"allocated":21499904,"data_stored":16726481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:21:21.375 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:21:21.375 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:21:21.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:21.477+0000 7f32a722b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:21.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:21.489+0000 7f32a722b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:21.657 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:21:21.657 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":715,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:21:20.721560+0000","last_up_change":"2024-03-30T13:21:04.538494+0000","last_in_change":"2024-03-30T13:21:04.538494+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":35,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":694,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":461,"last_clean_end":575,"up_from":581,"up_thru":646,"down_at":576,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6818","nonce":4030404750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6821","nonce":4030404750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6828","nonce":4030404750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6825","nonce":4030404750}]},"public_addr":"172.21.15.67:6818/4030404750","cluster_addr":"172.21.15.67:6821/4030404750","heartbeat_back_addr":"172.21.15.67:6828/4030404750","heartbeat_front_addr":"172.21.15.67:6825/4030404750","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":668,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":393,"last_clean_end":693,"up_from":699,"up_thru":699,"down_at":694,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1679277090},{"type":"v1","addr":"172.21.15.67:6801","nonce":1679277090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1679277090},{"type":"v1","addr":"172.21.15.67:6803","nonce":1679277090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1679277090},{"type":"v1","addr":"172.21.15.67:6807","nonce":1679277090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1679277090},{"type":"v1","addr":"172.21.15.67:6805","nonce":1679277090}]},"public_addr":"172.21.15.67:6801/1679277090","cluster_addr":"172.21.15.67:6803/1679277090","heartbeat_back_addr":"172.21.15.67:6807/1679277090","heartbeat_front_addr":"172.21.15.67:6805/1679277090","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":559,"last_clean_end":666,"up_from":677,"up_thru":699,"down_at":667,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1606011629},{"type":"v1","addr":"172.21.15.82:6803","nonce":1606011629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":1606011629},{"type":"v1","addr":"172.21.15.82:6807","nonce":1606011629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":1606011629},{"type":"v1","addr":"172.21.15.82:6815","nonce":1606011629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":1606011629},{"type":"v1","addr":"172.21.15.82:6811","nonce":1606011629}]},"public_addr":"172.21.15.82:6803/1606011629","cluster_addr":"172.21.15.82:6807/1606011629","heartbeat_back_addr":"172.21.15.82:6815/1606011629","heartbeat_front_addr":"172.21.15.82:6811/1606011629","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":488,"last_clean_end":506,"up_from":515,"up_thru":699,"down_at":507,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6817","nonce":292505453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6819","nonce":292505453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6823","nonce":292505453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6821","nonce":292505453}]},"public_addr":"172.21.15.82:6817/292505453","cluster_addr":"172.21.15.82:6819/292505453","heartbeat_back_addr":"172.21.15.82:6823/292505453","heartbeat_front_addr":"172.21.15.82:6821/292505453","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":372,"last_clean_end":621,"up_from":629,"up_thru":699,"down_at":622,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":3888194455},{"type":"v1","addr":"172.21.15.82:6801","nonce":3888194455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":3888194455},{"type":"v1","addr":"172.21.15.82:6806","nonce":3888194455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3888194455},{"type":"v1","addr":"172.21.15.82:6814","nonce":3888194455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3888194455},{"type":"v1","addr":"172.21.15.82:6810","nonce":3888194455}]},"public_addr":"172.21.15.82:6801/3888194455","cluster_addr":"172.21.15.82:6806/3888194455","heartbeat_back_addr":"172.21.15.82:6814/3888194455","heartbeat_front_addr":"172.21.15.82:6810/3888194455","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":604,"last_clean_end":645,"up_from":650,"up_thru":699,"down_at":646,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6825","nonce":1497758647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6827","nonce":1497758647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6831","nonce":1497758647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6829","nonce":1497758647}]},"public_addr":"172.21.15.82:6825/1497758647","cluster_addr":"172.21.15.82:6827/1497758647","heartbeat_back_addr":"172.21.15.82:6831/1497758647","heartbeat_front_addr":"172.21.15.82:6829/1497758647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:20:04.387532+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:20:08.907414+0000","dead_epoch":576},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:20:59.842948+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:21:06.735033+0000","dead_epoch":694},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:20:24.611629+0000","dead_epoch":667},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:21:14.902871+0000","dead_epoch":507},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:21:08.787068+0000","dead_epoch":622},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:20:32.719462+0000","dead_epoch":646}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:21:21.676 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:21:21.676 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:21:21.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:21.793+0000 7fad24def640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:21.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:21.809+0000 7fad24def640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:21.980 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:21:21.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:21:22.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:22.117+0000 7f806e252640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:22.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:22.133+0000 7f806e252640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:22.315 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:21:22.315 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:21:22.335 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:21:22.335 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:21:22.335 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:21:22.335 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:21:22.335 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 5, 1, 6, 7, 4, 3] 2024-03-30T13:21:22.335 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:21:22.335 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:21:22.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:22.453+0000 7f616385b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:22.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:22.465+0000 7f616385b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:22.641 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:21:22.641 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:21:22.659 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1483,"stamp":"2024-03-30T13:21:22.230977+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":251429354,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":455132,"kb_used_data":162452,"kb_used_omap":65,"kb_used_meta":292478,"kb_avail":754519588,"statfs":{"total":773094113280,"available":772628058112,"internally_reserved":0,"allocated":166350848,"data_stored":128163414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67493,"internal_metadata":299497563},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.542695"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1662,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-03-30T13:21:20.751768+0000","last_change":"2024-03-30T13:20:14.433776+0000","last_active":"2024-03-30T13:21:20.751768+0000","last_peered":"2024-03-30T13:21:20.751768+0000","last_clean":"2024-03-30T13:21:20.751768+0000","last_became_active":"2024-03-30T13:20:14.431136+0000","last_became_peered":"2024-03-30T13:20:14.431136+0000","last_unstale":"2024-03-30T13:21:20.751768+0000","last_undegraded":"2024-03-30T13:21:20.751768+0000","last_fullsized":"2024-03-30T13:21:20.751768+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:02.484916+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:20:02.484916+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:25.338815+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1508,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-03-30T13:21:17.690118+0000","last_change":"2024-03-30T13:21:17.333015+0000","last_active":"2024-03-30T13:21:17.690118+0000","last_peered":"2024-03-30T13:21:17.690118+0000","last_clean":"2024-03-30T13:21:17.690118+0000","last_became_active":"2024-03-30T13:21:05.571151+0000","last_became_peered":"2024-03-30T13:21:05.571151+0000","last_unstale":"2024-03-30T13:21:17.690118+0000","last_undegraded":"2024-03-30T13:21:17.690118+0000","last_fullsized":"2024-03-30T13:21:17.690118+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:17.332777+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:21:17.332777+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:44.049961+0000","scrub_duration":695,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":268,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-03-30T13:21:17.328486+0000","last_change":"2024-03-30T13:20:39.901251+0000","last_active":"2024-03-30T13:21:17.328486+0000","last_peered":"2024-03-30T13:21:17.328486+0000","last_clean":"2024-03-30T13:21:17.328486+0000","last_became_active":"2024-03-30T13:20:14.422234+0000","last_became_peered":"2024-03-30T13:20:14.422234+0000","last_unstale":"2024-03-30T13:21:17.328486+0000","last_undegraded":"2024-03-30T13:21:17.328486+0000","last_fullsized":"2024-03-30T13:21:17.328486+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:20:39.900992+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:20:39.900992+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:05.511322+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":256,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-03-30T13:21:17.328389+0000","last_change":"2024-03-30T13:20:10.775281+0000","last_active":"2024-03-30T13:21:17.328389+0000","last_peered":"2024-03-30T13:21:17.328389+0000","last_clean":"2024-03-30T13:21:17.328389+0000","last_became_active":"2024-03-30T13:19:23.949921+0000","last_became_peered":"2024-03-30T13:19:23.949921+0000","last_unstale":"2024-03-30T13:21:17.328389+0000","last_undegraded":"2024-03-30T13:21:17.328389+0000","last_fullsized":"2024-03-30T13:21:17.328389+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:10.775032+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:20:10.775032+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:18.932592+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":451,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-03-30T13:21:16.599612+0000","last_change":"2024-03-30T13:20:25.690889+0000","last_active":"2024-03-30T13:21:16.599612+0000","last_peered":"2024-03-30T13:21:16.599612+0000","last_clean":"2024-03-30T13:21:16.599612+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:21:16.599612+0000","last_undegraded":"2024-03-30T13:21:16.599612+0000","last_fullsized":"2024-03-30T13:21:16.599612+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:20:25.690651+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:55.652556+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1579,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-03-30T13:21:18.710485+0000","last_change":"2024-03-30T13:21:05.574895+0000","last_active":"2024-03-30T13:21:18.710485+0000","last_peered":"2024-03-30T13:21:18.710485+0000","last_clean":"2024-03-30T13:21:18.710485+0000","last_became_active":"2024-03-30T13:21:05.574411+0000","last_became_peered":"2024-03-30T13:21:05.574411+0000","last_unstale":"2024-03-30T13:21:18.710485+0000","last_undegraded":"2024-03-30T13:21:18.710485+0000","last_fullsized":"2024-03-30T13:21:18.710485+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:20:04.493986+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_clean_scrub_stamp":"2024-03-30T13:20:04.493986+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:20.538548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":316,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-03-30T13:21:17.328431+0000","last_change":"2024-03-30T13:20:45.976822+0000","last_active":"2024-03-30T13:21:17.328431+0000","last_peered":"2024-03-30T13:21:17.328431+0000","last_clean":"2024-03-30T13:21:17.328431+0000","last_became_active":"2024-03-30T13:20:14.421760+0000","last_became_peered":"2024-03-30T13:20:14.421760+0000","last_unstale":"2024-03-30T13:21:17.328431+0000","last_undegraded":"2024-03-30T13:21:17.328431+0000","last_fullsized":"2024-03-30T13:21:17.328431+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:20:45.976377+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:20:45.976377+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:56.308455+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":872,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-03-30T13:21:19.714145+0000","last_change":"2024-03-30T13:20:23.372796+0000","last_active":"2024-03-30T13:21:19.714145+0000","last_peered":"2024-03-30T13:21:19.714145+0000","last_clean":"2024-03-30T13:21:19.714145+0000","last_became_active":"2024-03-30T13:18:32.413904+0000","last_became_peered":"2024-03-30T13:18:32.413904+0000","last_unstale":"2024-03-30T13:21:19.714145+0000","last_undegraded":"2024-03-30T13:21:19.714145+0000","last_fullsized":"2024-03-30T13:21:19.714145+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:20:23.372460+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:20:23.372460+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:29.313113+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1718,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-03-30T13:21:18.710404+0000","last_change":"2024-03-30T13:21:05.574623+0000","last_active":"2024-03-30T13:21:18.710404+0000","last_peered":"2024-03-30T13:21:18.710404+0000","last_clean":"2024-03-30T13:21:18.710404+0000","last_became_active":"2024-03-30T13:21:05.574131+0000","last_became_peered":"2024-03-30T13:21:05.574131+0000","last_unstale":"2024-03-30T13:21:18.710404+0000","last_undegraded":"2024-03-30T13:21:18.710404+0000","last_fullsized":"2024-03-30T13:21:18.710404+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:56.407671+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:20:56.407671+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:11.138773+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1330,"reported_epoch":712,"state":"active+clean","last_fresh":"2024-03-30T13:21:17.690161+0000","last_change":"2024-03-30T13:20:23.631457+0000","last_active":"2024-03-30T13:21:17.690161+0000","last_peered":"2024-03-30T13:21:17.690161+0000","last_clean":"2024-03-30T13:21:17.690161+0000","last_became_active":"2024-03-30T13:20:14.427056+0000","last_became_peered":"2024-03-30T13:20:14.427056+0000","last_unstale":"2024-03-30T13:21:17.690161+0000","last_undegraded":"2024-03-30T13:21:17.690161+0000","last_fullsized":"2024-03-30T13:21:17.690161+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:20:23.631067+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:42.480355+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1645,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-03-30T13:21:20.751366+0000","last_change":"2024-03-30T13:20:42.398798+0000","last_active":"2024-03-30T13:21:20.751366+0000","last_peered":"2024-03-30T13:21:20.751366+0000","last_clean":"2024-03-30T13:21:20.751366+0000","last_became_active":"2024-03-30T13:20:14.431380+0000","last_became_peered":"2024-03-30T13:20:14.431380+0000","last_unstale":"2024-03-30T13:21:20.751366+0000","last_undegraded":"2024-03-30T13:21:20.751366+0000","last_fullsized":"2024-03-30T13:21:20.751366+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:20:42.398509+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:53.484365+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1702,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-03-30T13:21:20.751624+0000","last_change":"2024-03-30T13:21:05.572623+0000","last_active":"2024-03-30T13:21:20.751624+0000","last_peered":"2024-03-30T13:21:20.751624+0000","last_clean":"2024-03-30T13:21:20.751624+0000","last_became_active":"2024-03-30T13:21:05.570917+0000","last_became_peered":"2024-03-30T13:21:05.570917+0000","last_unstale":"2024-03-30T13:21:20.751624+0000","last_undegraded":"2024-03-30T13:21:20.751624+0000","last_fullsized":"2024-03-30T13:21:20.751624+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:20:35.455810+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:20:35.455810+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:00.400375+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":485,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-03-30T13:21:16.599176+0000","last_change":"2024-03-30T13:21:05.573717+0000","last_active":"2024-03-30T13:21:16.599176+0000","last_peered":"2024-03-30T13:21:16.599176+0000","last_clean":"2024-03-30T13:21:16.599176+0000","last_became_active":"2024-03-30T13:21:05.572577+0000","last_became_peered":"2024-03-30T13:21:05.572577+0000","last_unstale":"2024-03-30T13:21:16.599176+0000","last_undegraded":"2024-03-30T13:21:16.599176+0000","last_fullsized":"2024-03-30T13:21:16.599176+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:20:28.696431+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:20:28.696431+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:31.562367+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":270,"reported_epoch":711,"state":"active+clean","last_fresh":"2024-03-30T13:21:17.328481+0000","last_change":"2024-03-30T13:21:06.773969+0000","last_active":"2024-03-30T13:21:17.328481+0000","last_peered":"2024-03-30T13:21:17.328481+0000","last_clean":"2024-03-30T13:21:17.328481+0000","last_became_active":"2024-03-30T13:21:05.570131+0000","last_became_peered":"2024-03-30T13:21:05.570131+0000","last_unstale":"2024-03-30T13:21:17.328481+0000","last_undegraded":"2024-03-30T13:21:17.328481+0000","last_fullsized":"2024-03-30T13:21:17.328481+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:06.773612+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:21:06.773612+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:19.252741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":470,"reported_epoch":710,"state":"active+clean","last_fresh":"2024-03-30T13:21:16.599658+0000","last_change":"2024-03-30T13:20:54.701831+0000","last_active":"2024-03-30T13:21:16.599658+0000","last_peered":"2024-03-30T13:21:16.599658+0000","last_clean":"2024-03-30T13:21:16.599658+0000","last_became_active":"2024-03-30T13:18:32.415413+0000","last_became_peered":"2024-03-30T13:18:32.415413+0000","last_unstale":"2024-03-30T13:21:16.599658+0000","last_undegraded":"2024-03-30T13:21:16.599658+0000","last_fullsized":"2024-03-30T13:21:16.599658+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:20:54.701569+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:20:54.701569+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:18.695855+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":833,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-03-30T13:21:19.714328+0000","last_change":"2024-03-30T13:20:41.484219+0000","last_active":"2024-03-30T13:21:19.714328+0000","last_peered":"2024-03-30T13:21:19.714328+0000","last_clean":"2024-03-30T13:21:19.714328+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:21:19.714328+0000","last_undegraded":"2024-03-30T13:21:19.714328+0000","last_fullsized":"2024-03-30T13:21:19.714328+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:41.483960+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:20:41.483960+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:59.137796+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":251429354,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":629,"seq":2701534429220,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60396,"kb_used_data":23640,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94311444,"statfs":{"total":96636764160,"available":96574918656,"internally_reserved":0,"allocated":24207360,"data_stored":19413534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157522,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65960,"kb_used_data":28312,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94305880,"statfs":{"total":96636764160,"available":96569221120,"internally_reserved":0,"allocated":28991488,"data_stored":24238418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":677,"seq":2907692859407,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57692,"kb_used_data":20040,"kb_used_omap":13,"kb_used_meta":37618,"kb_avail":94314148,"statfs":{"total":96636764160,"available":96577687552,"internally_reserved":0,"allocated":20520960,"data_stored":15754338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14287,"internal_metadata":38520881},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509753,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50176,"kb_used_data":14832,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94321664,"statfs":{"total":96636764160,"available":96585383936,"internally_reserved":0,"allocated":15187968,"data_stored":10407744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375999032,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51312,"kb_used_data":14108,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94320528,"statfs":{"total":96636764160,"available":96584220672,"internally_reserved":0,"allocated":14446592,"data_stored":9666368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509754,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57356,"kb_used_data":22008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94314484,"statfs":{"total":96636764160,"available":96578031616,"internally_reserved":0,"allocated":22536192,"data_stored":17755968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742426,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59104,"kb_used_data":18516,"kb_used_omap":15,"kb_used_meta":40560,"kb_avail":94312736,"statfs":{"total":96636764160,"available":96576241664,"internally_reserved":0,"allocated":18960384,"data_stored":14200563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":41533891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":699,"seq":3002182139909,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53136,"kb_used_data":20996,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94318704,"statfs":{"total":96636764160,"available":96582352896,"internally_reserved":0,"allocated":21499904,"data_stored":16726481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:21:22.660 INFO:tasks.thrashosds.thrasher:Choosing PG 1.f with acting set [6, 5, 4] 2024-03-30T13:21:22.660 INFO:tasks.thrashosds.thrasher:Killing osd 6, live_osds are [0, 2, 5, 1, 6, 7, 4, 3] 2024-03-30T13:21:22.661 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-30T13:21:22.661 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:21:22.726 INFO:tasks.ceph.osd.6:Stopped 2024-03-30T13:21:22.726 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 2, 5, 1, 6, 7, 4, 3] 2024-03-30T13:21:22.726 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 6 2024-03-30T13:21:22.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:22.849+0000 7f274b9d7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:22.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:22.865+0000 7f274b9d7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:23.038 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:21:23.037+0000 7fbb38f08640 -1 mon.a@0(leader).osd e716 definitely_dead 0 2024-03-30T13:21:23.750 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.6. 2024-03-30T13:21:23.780 INFO:tasks.thrashosds.thrasher:dead_osds=[6], live_osds=[0, 2, 5, 1, 7, 4, 3] 2024-03-30T13:21:23.780 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:21:23.781 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:21:23.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:23.901+0000 7f82889ed640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:23.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:23.913+0000 7f82889ed640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:24.084 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:21:24.085 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:21:24.103 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1485,"stamp":"2024-03-30T13:21:23.747468+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":251429354,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":41,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":394932,"kb_used_data":138992,"kb_used_omap":58,"kb_used_meta":255749,"kb_avail":660207948,"statfs":{"total":676457349120,"available":676052938752,"internally_reserved":0,"allocated":142327808,"data_stored":108832235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59799,"internal_metadata":261887593},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.040149"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1662,"reported_epoch":715,"state":"stale+active+clean","last_fresh":"2024-03-30T13:21:20.751768+0000","last_change":"2024-03-30T13:20:14.433776+0000","last_active":"2024-03-30T13:21:20.751768+0000","last_peered":"2024-03-30T13:21:20.751768+0000","last_clean":"2024-03-30T13:21:20.751768+0000","last_became_active":"2024-03-30T13:20:14.431136+0000","last_became_peered":"2024-03-30T13:20:14.431136+0000","last_unstale":"2024-03-30T13:21:22.745044+0000","last_undegraded":"2024-03-30T13:21:20.751768+0000","last_fullsized":"2024-03-30T13:21:20.751768+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:02.484916+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:20:02.484916+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:25.338815+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1516,"reported_epoch":716,"state":"active+clean","last_fresh":"2024-03-30T13:21:22.748591+0000","last_change":"2024-03-30T13:21:17.333015+0000","last_active":"2024-03-30T13:21:22.748591+0000","last_peered":"2024-03-30T13:21:22.748591+0000","last_clean":"2024-03-30T13:21:22.748591+0000","last_became_active":"2024-03-30T13:21:05.571151+0000","last_became_peered":"2024-03-30T13:21:05.571151+0000","last_unstale":"2024-03-30T13:21:22.748591+0000","last_undegraded":"2024-03-30T13:21:22.748591+0000","last_fullsized":"2024-03-30T13:21:22.748591+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:17.332777+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:21:17.332777+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:44.049961+0000","scrub_duration":695,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":276,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-03-30T13:21:20.751390+0000","last_change":"2024-03-30T13:20:39.901251+0000","last_active":"2024-03-30T13:21:20.751390+0000","last_peered":"2024-03-30T13:21:20.751390+0000","last_clean":"2024-03-30T13:21:20.751390+0000","last_became_active":"2024-03-30T13:20:14.422234+0000","last_became_peered":"2024-03-30T13:20:14.422234+0000","last_unstale":"2024-03-30T13:21:20.751390+0000","last_undegraded":"2024-03-30T13:21:20.751390+0000","last_fullsized":"2024-03-30T13:21:20.751390+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:20:39.900992+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:20:39.900992+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:05.511322+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":265,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-03-30T13:21:20.751382+0000","last_change":"2024-03-30T13:20:10.775281+0000","last_active":"2024-03-30T13:21:20.751382+0000","last_peered":"2024-03-30T13:21:20.751382+0000","last_clean":"2024-03-30T13:21:20.751382+0000","last_became_active":"2024-03-30T13:19:23.949921+0000","last_became_peered":"2024-03-30T13:19:23.949921+0000","last_unstale":"2024-03-30T13:21:20.751382+0000","last_undegraded":"2024-03-30T13:21:20.751382+0000","last_fullsized":"2024-03-30T13:21:20.751382+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:10.775032+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:09:53.319978+0000","last_clean_scrub_stamp":"2024-03-30T13:20:10.775032+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2168070,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":461,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-03-30T13:21:20.750211+0000","last_change":"2024-03-30T13:20:25.690889+0000","last_active":"2024-03-30T13:21:20.750211+0000","last_peered":"2024-03-30T13:21:20.750211+0000","last_clean":"2024-03-30T13:21:20.750211+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:21:20.750211+0000","last_undegraded":"2024-03-30T13:21:20.750211+0000","last_fullsized":"2024-03-30T13:21:20.750211+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:20:25.690651+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:55.652556+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1579,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-03-30T13:21:18.710485+0000","last_change":"2024-03-30T13:21:05.574895+0000","last_active":"2024-03-30T13:21:18.710485+0000","last_peered":"2024-03-30T13:21:18.710485+0000","last_clean":"2024-03-30T13:21:18.710485+0000","last_became_active":"2024-03-30T13:21:05.574411+0000","last_became_peered":"2024-03-30T13:21:05.574411+0000","last_unstale":"2024-03-30T13:21:18.710485+0000","last_undegraded":"2024-03-30T13:21:18.710485+0000","last_fullsized":"2024-03-30T13:21:18.710485+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:20:04.493986+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:18:42.729016+0000","last_clean_scrub_stamp":"2024-03-30T13:20:04.493986+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:20.538548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":324,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-03-30T13:21:20.751288+0000","last_change":"2024-03-30T13:20:45.976822+0000","last_active":"2024-03-30T13:21:20.751288+0000","last_peered":"2024-03-30T13:21:20.751288+0000","last_clean":"2024-03-30T13:21:20.751288+0000","last_became_active":"2024-03-30T13:20:14.421760+0000","last_became_peered":"2024-03-30T13:20:14.421760+0000","last_unstale":"2024-03-30T13:21:20.751288+0000","last_undegraded":"2024-03-30T13:21:20.751288+0000","last_fullsized":"2024-03-30T13:21:20.751288+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:20:45.976377+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:20:45.976377+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:56.308455+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":872,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-03-30T13:21:19.714145+0000","last_change":"2024-03-30T13:20:23.372796+0000","last_active":"2024-03-30T13:21:19.714145+0000","last_peered":"2024-03-30T13:21:19.714145+0000","last_clean":"2024-03-30T13:21:19.714145+0000","last_became_active":"2024-03-30T13:18:32.413904+0000","last_became_peered":"2024-03-30T13:18:32.413904+0000","last_unstale":"2024-03-30T13:21:19.714145+0000","last_undegraded":"2024-03-30T13:21:19.714145+0000","last_fullsized":"2024-03-30T13:21:19.714145+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:20:23.372460+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:20:23.372460+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:29.313113+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1718,"reported_epoch":713,"state":"active+clean","last_fresh":"2024-03-30T13:21:18.710404+0000","last_change":"2024-03-30T13:21:05.574623+0000","last_active":"2024-03-30T13:21:18.710404+0000","last_peered":"2024-03-30T13:21:18.710404+0000","last_clean":"2024-03-30T13:21:18.710404+0000","last_became_active":"2024-03-30T13:21:05.574131+0000","last_became_peered":"2024-03-30T13:21:05.574131+0000","last_unstale":"2024-03-30T13:21:18.710404+0000","last_undegraded":"2024-03-30T13:21:18.710404+0000","last_fullsized":"2024-03-30T13:21:18.710404+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:56.407671+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:20:56.407671+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:11.138773+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1337,"reported_epoch":715,"state":"peering","last_fresh":"2024-03-30T13:21:20.728484+0000","last_change":"2024-03-30T13:21:22.748365+0000","last_active":"2024-03-30T13:21:22.748365+0000","last_peered":"2024-03-30T13:21:20.728484+0000","last_clean":"2024-03-30T13:21:20.728484+0000","last_became_active":"2024-03-30T13:20:14.427056+0000","last_became_peered":"2024-03-30T13:20:14.427056+0000","last_unstale":"2024-03-30T13:21:20.728484+0000","last_undegraded":"2024-03-30T13:21:20.728484+0000","last_fullsized":"2024-03-30T13:21:20.728484+0000","mapping_epoch":716,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:20:23.631067+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,2147483647],"acting":[4,2,2147483647],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1645,"reported_epoch":715,"state":"stale+active+clean","last_fresh":"2024-03-30T13:21:20.751366+0000","last_change":"2024-03-30T13:20:42.398798+0000","last_active":"2024-03-30T13:21:20.751366+0000","last_peered":"2024-03-30T13:21:20.751366+0000","last_clean":"2024-03-30T13:21:20.751366+0000","last_became_active":"2024-03-30T13:20:14.431380+0000","last_became_peered":"2024-03-30T13:20:14.431380+0000","last_unstale":"2024-03-30T13:21:22.745053+0000","last_undegraded":"2024-03-30T13:21:20.751366+0000","last_fullsized":"2024-03-30T13:21:20.751366+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:20:42.398509+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:53.484365+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1702,"reported_epoch":715,"state":"stale+active+clean","last_fresh":"2024-03-30T13:21:20.751624+0000","last_change":"2024-03-30T13:21:05.572623+0000","last_active":"2024-03-30T13:21:20.751624+0000","last_peered":"2024-03-30T13:21:20.751624+0000","last_clean":"2024-03-30T13:21:20.751624+0000","last_became_active":"2024-03-30T13:21:05.570917+0000","last_became_peered":"2024-03-30T13:21:05.570917+0000","last_unstale":"2024-03-30T13:21:22.745057+0000","last_undegraded":"2024-03-30T13:21:20.751624+0000","last_fullsized":"2024-03-30T13:21:20.751624+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:20:35.455810+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:20:35.455810+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:00.400375+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":495,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-03-30T13:21:20.750215+0000","last_change":"2024-03-30T13:21:05.573717+0000","last_active":"2024-03-30T13:21:20.750215+0000","last_peered":"2024-03-30T13:21:20.750215+0000","last_clean":"2024-03-30T13:21:20.750215+0000","last_became_active":"2024-03-30T13:21:05.572577+0000","last_became_peered":"2024-03-30T13:21:05.572577+0000","last_unstale":"2024-03-30T13:21:20.750215+0000","last_undegraded":"2024-03-30T13:21:20.750215+0000","last_fullsized":"2024-03-30T13:21:20.750215+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:20:28.696431+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:20:28.696431+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:31.562367+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":278,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-03-30T13:21:20.751581+0000","last_change":"2024-03-30T13:21:06.773969+0000","last_active":"2024-03-30T13:21:20.751581+0000","last_peered":"2024-03-30T13:21:20.751581+0000","last_clean":"2024-03-30T13:21:20.751581+0000","last_became_active":"2024-03-30T13:21:05.570131+0000","last_became_peered":"2024-03-30T13:21:05.570131+0000","last_unstale":"2024-03-30T13:21:20.751581+0000","last_undegraded":"2024-03-30T13:21:20.751581+0000","last_fullsized":"2024-03-30T13:21:20.751581+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:06.773612+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:21:06.773612+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:19.252741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":480,"reported_epoch":715,"state":"active+clean","last_fresh":"2024-03-30T13:21:20.750315+0000","last_change":"2024-03-30T13:20:54.701831+0000","last_active":"2024-03-30T13:21:20.750315+0000","last_peered":"2024-03-30T13:21:20.750315+0000","last_clean":"2024-03-30T13:21:20.750315+0000","last_became_active":"2024-03-30T13:18:32.415413+0000","last_became_peered":"2024-03-30T13:18:32.415413+0000","last_unstale":"2024-03-30T13:21:20.750315+0000","last_undegraded":"2024-03-30T13:21:20.750315+0000","last_fullsized":"2024-03-30T13:21:20.750315+0000","mapping_epoch":629,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":630,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:20:54.701569+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:20:54.701569+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:18.695855+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":833,"reported_epoch":714,"state":"active+clean","last_fresh":"2024-03-30T13:21:19.714328+0000","last_change":"2024-03-30T13:20:41.484219+0000","last_active":"2024-03-30T13:21:19.714328+0000","last_peered":"2024-03-30T13:21:19.714328+0000","last_clean":"2024-03-30T13:21:19.714328+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:21:19.714328+0000","last_undegraded":"2024-03-30T13:21:19.714328+0000","last_fullsized":"2024-03-30T13:21:19.714328+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:41.483960+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:20:41.483960+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:59.137796+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":102,"num_bytes_recovered":251429354,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157523,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66036,"kb_used_data":28384,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94305804,"statfs":{"total":96636764160,"available":96569143296,"internally_reserved":0,"allocated":29065216,"data_stored":24271390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":677,"seq":2907692859408,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57740,"kb_used_data":20088,"kb_used_omap":13,"kb_used_meta":37618,"kb_avail":94314100,"statfs":{"total":96636764160,"available":96577638400,"internally_reserved":0,"allocated":20570112,"data_stored":15776286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14287,"internal_metadata":38520881},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509753,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50176,"kb_used_data":14832,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94321664,"statfs":{"total":96636764160,"available":96585383936,"internally_reserved":0,"allocated":15187968,"data_stored":10407744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375999032,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51312,"kb_used_data":14108,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94320528,"statfs":{"total":96636764160,"available":96584220672,"internally_reserved":0,"allocated":14446592,"data_stored":9666368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509754,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57356,"kb_used_data":22008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94314484,"statfs":{"total":96636764160,"available":96578031616,"internally_reserved":0,"allocated":22536192,"data_stored":17755968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742427,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59176,"kb_used_data":18576,"kb_used_omap":15,"kb_used_meta":40560,"kb_avail":94312664,"statfs":{"total":96636764160,"available":96576167936,"internally_reserved":0,"allocated":19021824,"data_stored":14227998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":41533891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":699,"seq":3002182139909,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53136,"kb_used_data":20996,"kb_used_omap":5,"kb_used_meta":32122,"kb_avail":94318704,"statfs":{"total":96636764160,"available":96582352896,"internally_reserved":0,"allocated":21499904,"data_stored":16726481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":32893026},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:21:24.104 INFO:tasks.ceph.ceph_manager.ceph:PG 1.5 is not active or peered 2024-03-30T13:21:24.104 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.5', 'version': "22'3", 'reported_seq': 1337, 'reported_epoch': 715, 'state': 'peering', 'last_fresh': '2024-03-30T13:21:20.728484+0000', 'last_change': '2024-03-30T13:21:22.748365+0000', 'last_active': '2024-03-30T13:21:22.748365+0000', 'last_peered': '2024-03-30T13:21:20.728484+0000', 'last_clean': '2024-03-30T13:21:20.728484+0000', 'last_became_active': '2024-03-30T13:20:14.427056+0000', 'last_became_peered': '2024-03-30T13:20:14.427056+0000', 'last_unstale': '2024-03-30T13:21:20.728484+0000', 'last_undegraded': '2024-03-30T13:21:20.728484+0000', 'last_fullsized': '2024-03-30T13:21:20.728484+0000', 'mapping_epoch': 716, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 19, 'last_epoch_clean': 678, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "22'3", 'last_scrub_stamp': '2024-03-30T13:20:23.631067+0000', 'last_deep_scrub': "22'3", 'last_deep_scrub_stamp': '2024-03-30T13:20:23.631067+0000', 'last_clean_scrub_stamp': '2024-03-30T13:20:23.631067+0000', 'objects_scrubbed': 1, 'log_size': 3, 'log_dups_size': 0, 'ondisk_log_size': 3, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 7, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 2336140, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 3, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 1, 'num_read_kb': 0, 'num_write': 4, 'num_write_kb': 835, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 2336140, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2, 2147483647], 'acting': [4, 2, 2147483647], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-30T13:21:24.104 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:21:49.105 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:21:49.105 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:21:49.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:49.232+0000 7f9c64552640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:49.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:49.244+0000 7f9c64552640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:49.422 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:21:49.422 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:21:49.440 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1501,"stamp":"2024-03-30T13:21:48.238906+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":270185702,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":415836,"kb_used_data":155784,"kb_used_omap":58,"kb_used_meta":259909,"kb_avail":660187044,"statfs":{"total":676457349120,"available":676031533056,"internally_reserved":0,"allocated":159522816,"data_stored":125765371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59799,"internal_metadata":266147433},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002662"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":27,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-03-30T13:21:40.553529+0000","last_change":"2024-03-30T13:21:28.696659+0000","last_active":"2024-03-30T13:21:40.553529+0000","last_peered":"2024-03-30T13:21:40.553529+0000","last_clean":"2024-03-30T13:21:40.553529+0000","last_became_active":"2024-03-30T13:21:24.763304+0000","last_became_peered":"2024-03-30T13:21:24.763304+0000","last_unstale":"2024-03-30T13:21:40.553529+0000","last_undegraded":"2024-03-30T13:21:40.553529+0000","last_fullsized":"2024-03-30T13:21:40.553529+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:21:28.696384+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:21:28.696384+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:56.174122+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1522,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-03-30T13:21:40.554164+0000","last_change":"2024-03-30T13:21:17.333015+0000","last_active":"2024-03-30T13:21:40.554164+0000","last_peered":"2024-03-30T13:21:40.554164+0000","last_clean":"2024-03-30T13:21:40.554164+0000","last_became_active":"2024-03-30T13:21:05.571151+0000","last_became_peered":"2024-03-30T13:21:05.571151+0000","last_unstale":"2024-03-30T13:21:40.554164+0000","last_undegraded":"2024-03-30T13:21:40.554164+0000","last_fullsized":"2024-03-30T13:21:40.554164+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:17.332777+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:21:17.332777+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:44.049961+0000","scrub_duration":695,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":287,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-03-30T13:21:45.711746+0000","last_change":"2024-03-30T13:20:39.901251+0000","last_active":"2024-03-30T13:21:45.711746+0000","last_peered":"2024-03-30T13:21:45.711746+0000","last_clean":"2024-03-30T13:21:45.711746+0000","last_became_active":"2024-03-30T13:20:14.422234+0000","last_became_peered":"2024-03-30T13:20:14.422234+0000","last_unstale":"2024-03-30T13:21:45.711746+0000","last_undegraded":"2024-03-30T13:21:45.711746+0000","last_fullsized":"2024-03-30T13:21:45.711746+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:20:39.900992+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:08:37.747009+0000","last_clean_scrub_stamp":"2024-03-30T13:20:39.900992+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:05.511322+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":294,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-03-30T13:21:45.711666+0000","last_change":"2024-03-30T13:21:26.828956+0000","last_active":"2024-03-30T13:21:45.711666+0000","last_peered":"2024-03-30T13:21:45.711666+0000","last_clean":"2024-03-30T13:21:45.711666+0000","last_became_active":"2024-03-30T13:21:24.767670+0000","last_became_peered":"2024-03-30T13:21:24.767670+0000","last_unstale":"2024-03-30T13:21:45.711666+0000","last_undegraded":"2024-03-30T13:21:45.711666+0000","last_fullsized":"2024-03-30T13:21:45.711666+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:21:26.828701+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:21:26.828701+0000","last_clean_scrub_stamp":"2024-03-30T13:21:26.828701+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:36.093272+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":471,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-03-30T13:21:45.475008+0000","last_change":"2024-03-30T13:20:25.690889+0000","last_active":"2024-03-30T13:21:45.475008+0000","last_peered":"2024-03-30T13:21:45.475008+0000","last_clean":"2024-03-30T13:21:45.475008+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:21:45.475008+0000","last_undegraded":"2024-03-30T13:21:45.475008+0000","last_fullsized":"2024-03-30T13:21:45.475008+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:20:25.690651+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:55.652556+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1599,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-03-30T13:21:40.553561+0000","last_change":"2024-03-30T13:21:21.676616+0000","last_active":"2024-03-30T13:21:40.553561+0000","last_peered":"2024-03-30T13:21:40.553561+0000","last_clean":"2024-03-30T13:21:40.553561+0000","last_became_active":"2024-03-30T13:21:05.574411+0000","last_became_peered":"2024-03-30T13:21:05.574411+0000","last_unstale":"2024-03-30T13:21:40.553561+0000","last_undegraded":"2024-03-30T13:21:40.553561+0000","last_fullsized":"2024-03-30T13:21:40.553561+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:21:21.676444+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:29.360053+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":335,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-03-30T13:21:45.711701+0000","last_change":"2024-03-30T13:20:45.976822+0000","last_active":"2024-03-30T13:21:45.711701+0000","last_peered":"2024-03-30T13:21:45.711701+0000","last_clean":"2024-03-30T13:21:45.711701+0000","last_became_active":"2024-03-30T13:20:14.421760+0000","last_became_peered":"2024-03-30T13:20:14.421760+0000","last_unstale":"2024-03-30T13:21:45.711701+0000","last_undegraded":"2024-03-30T13:21:45.711701+0000","last_fullsized":"2024-03-30T13:21:45.711701+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:20:45.976377+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:20:45.976377+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:56.308455+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":903,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-03-30T13:21:40.553137+0000","last_change":"2024-03-30T13:21:30.546406+0000","last_active":"2024-03-30T13:21:40.553137+0000","last_peered":"2024-03-30T13:21:40.553137+0000","last_clean":"2024-03-30T13:21:40.553137+0000","last_became_active":"2024-03-30T13:21:24.767181+0000","last_became_peered":"2024-03-30T13:21:24.767181+0000","last_unstale":"2024-03-30T13:21:40.553137+0000","last_undegraded":"2024-03-30T13:21:40.553137+0000","last_fullsized":"2024-03-30T13:21:40.553137+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:21:30.546139+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:21:30.546139+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:50.975110+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":73445600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1730,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-03-30T13:21:40.553354+0000","last_change":"2024-03-30T13:21:05.574623+0000","last_active":"2024-03-30T13:21:40.553354+0000","last_peered":"2024-03-30T13:21:40.553354+0000","last_clean":"2024-03-30T13:21:40.553354+0000","last_became_active":"2024-03-30T13:21:05.574131+0000","last_became_peered":"2024-03-30T13:21:05.574131+0000","last_unstale":"2024-03-30T13:21:40.553354+0000","last_undegraded":"2024-03-30T13:21:40.553354+0000","last_fullsized":"2024-03-30T13:21:40.553354+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:56.407671+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:20:56.407671+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:11.138773+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1353,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-03-30T13:21:40.553864+0000","last_change":"2024-03-30T13:21:24.797086+0000","last_active":"2024-03-30T13:21:40.553864+0000","last_peered":"2024-03-30T13:21:40.553864+0000","last_clean":"2024-03-30T13:21:40.553864+0000","last_became_active":"2024-03-30T13:21:24.767072+0000","last_became_peered":"2024-03-30T13:21:24.767072+0000","last_unstale":"2024-03-30T13:21:40.553864+0000","last_undegraded":"2024-03-30T13:21:40.553864+0000","last_fullsized":"2024-03-30T13:21:40.553864+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:20:23.631067+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:41.944142+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":27,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-03-30T13:21:45.711763+0000","last_change":"2024-03-30T13:21:25.215384+0000","last_active":"2024-03-30T13:21:45.711763+0000","last_peered":"2024-03-30T13:21:45.711763+0000","last_clean":"2024-03-30T13:21:45.711763+0000","last_became_active":"2024-03-30T13:21:24.767636+0000","last_became_peered":"2024-03-30T13:21:24.767636+0000","last_unstale":"2024-03-30T13:21:45.711763+0000","last_undegraded":"2024-03-30T13:21:45.711763+0000","last_fullsized":"2024-03-30T13:21:45.711763+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:20:42.398509+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:05.489703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":33,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-03-30T13:21:45.711923+0000","last_change":"2024-03-30T13:21:24.932632+0000","last_active":"2024-03-30T13:21:45.711923+0000","last_peered":"2024-03-30T13:21:45.711923+0000","last_clean":"2024-03-30T13:21:45.711923+0000","last_became_active":"2024-03-30T13:21:24.765276+0000","last_became_peered":"2024-03-30T13:21:24.765276+0000","last_unstale":"2024-03-30T13:21:45.711923+0000","last_undegraded":"2024-03-30T13:21:45.711923+0000","last_fullsized":"2024-03-30T13:21:45.711923+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:20:35.455810+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:20:35.455810+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:48.125256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":513,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-03-30T13:21:45.475008+0000","last_change":"2024-03-30T13:21:33.418242+0000","last_active":"2024-03-30T13:21:45.475008+0000","last_peered":"2024-03-30T13:21:45.475008+0000","last_clean":"2024-03-30T13:21:45.475008+0000","last_became_active":"2024-03-30T13:21:05.572577+0000","last_became_peered":"2024-03-30T13:21:05.572577+0000","last_unstale":"2024-03-30T13:21:45.475008+0000","last_undegraded":"2024-03-30T13:21:45.475008+0000","last_fullsized":"2024-03-30T13:21:45.475008+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:21:33.417986+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:21:33.417986+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:41.633765+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":289,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-03-30T13:21:45.711541+0000","last_change":"2024-03-30T13:21:06.773969+0000","last_active":"2024-03-30T13:21:45.711541+0000","last_peered":"2024-03-30T13:21:45.711541+0000","last_clean":"2024-03-30T13:21:45.711541+0000","last_became_active":"2024-03-30T13:21:05.570131+0000","last_became_peered":"2024-03-30T13:21:05.570131+0000","last_unstale":"2024-03-30T13:21:45.711541+0000","last_undegraded":"2024-03-30T13:21:45.711541+0000","last_fullsized":"2024-03-30T13:21:45.711541+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:06.773612+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:21:06.773612+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:19.252741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":505,"reported_epoch":720,"state":"active+clean","last_fresh":"2024-03-30T13:21:45.475061+0000","last_change":"2024-03-30T13:21:24.971884+0000","last_active":"2024-03-30T13:21:45.475061+0000","last_peered":"2024-03-30T13:21:45.475061+0000","last_clean":"2024-03-30T13:21:45.475061+0000","last_became_active":"2024-03-30T13:21:24.764300+0000","last_became_peered":"2024-03-30T13:21:24.764300+0000","last_unstale":"2024-03-30T13:21:45.475061+0000","last_undegraded":"2024-03-30T13:21:45.475061+0000","last_fullsized":"2024-03-30T13:21:45.475061+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:20:54.701569+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:20:54.701569+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:14.516246+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":843,"reported_epoch":719,"state":"active+clean","last_fresh":"2024-03-30T13:21:40.553139+0000","last_change":"2024-03-30T13:20:41.484219+0000","last_active":"2024-03-30T13:21:40.553139+0000","last_peered":"2024-03-30T13:21:40.553139+0000","last_clean":"2024-03-30T13:21:40.553139+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:21:40.553139+0000","last_undegraded":"2024-03-30T13:21:40.553139+0000","last_fullsized":"2024-03-30T13:21:40.553139+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:41.483960+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:20:41.483960+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:21:59.137796+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":270185702,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":113467392,"data_stored":113467392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157528,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66088,"kb_used_data":28444,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94305752,"statfs":{"total":96636764160,"available":96569090048,"internally_reserved":0,"allocated":29126656,"data_stored":24299336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":677,"seq":2907692859412,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57780,"kb_used_data":20136,"kb_used_omap":13,"kb_used_meta":37618,"kb_avail":94314060,"statfs":{"total":96636764160,"available":96577597440,"internally_reserved":0,"allocated":20619264,"data_stored":15798703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14287,"internal_metadata":38520881},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509758,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51560,"kb_used_data":16220,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94320280,"statfs":{"total":96636764160,"available":96583966720,"internally_reserved":0,"allocated":16609280,"data_stored":11788719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375999037,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51376,"kb_used_data":14180,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94320464,"statfs":{"total":96636764160,"available":96584155136,"internally_reserved":0,"allocated":14520320,"data_stored":9699759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509759,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62520,"kb_used_data":27180,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94309320,"statfs":{"total":96636764160,"available":96572743680,"internally_reserved":0,"allocated":27832320,"data_stored":23011759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742432,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68104,"kb_used_data":27516,"kb_used_omap":15,"kb_used_meta":40560,"kb_avail":94303736,"statfs":{"total":96636764160,"available":96567025664,"internally_reserved":0,"allocated":28176384,"data_stored":23349064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":41533891},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":699,"seq":3002182139914,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58408,"kb_used_data":22108,"kb_used_omap":5,"kb_used_meta":36282,"kb_avail":94313432,"statfs":{"total":96636764160,"available":96576954368,"internally_reserved":0,"allocated":22638592,"data_stored":17818031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":37152866},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19046400,"data_stored":19046400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7823360,"data_stored":7823360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":13852672,"data_stored":13852672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":19378176,"data_stored":19378176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:21:49.441 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:21:49.441 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:21:49.441 INFO:tasks.thrashosds.thrasher:Reviving osd 6 2024-03-30T13:21:49.441 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-30T13:21:49.441 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-30T13:21:49.444 INFO:tasks.ceph.osd.6:Started 2024-03-30T13:21:49.444 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-30T13:21:49.514 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T13:21:49.514+0000 7f379213c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:49.522 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T13:21:49.522+0000 7f379213c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:49.522 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T13:21:49.522+0000 7f379213c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:49.569 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T13:21:49.569 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T13:21:49.569 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T13:21:49.569 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T13:21:49.569 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T13:21:49.569 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T13:21:49.569 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-03-30T13:21:49.569 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-03-30T13:21:49.570 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-03-30T13:21:49.570 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-03-30T13:21:49.570 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-03-30T13:21:49.570 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T13:21:49.571 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:21:49.572 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-03-30T13:21:50.062 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T13:21:50.062+0000 7f379213c7c0 -1 Falling back to public interface 2024-03-30T13:21:53.862 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T13:21:53.862+0000 7f379213c7c0 -1 osd.6 715 log_to_monitors true 2024-03-30T13:21:54.545 INFO:tasks.ceph.osd.6.smithi082.stderr:2024-03-30T13:21:54.546+0000 7f3789f11640 -1 osd.6 715 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:21:54.572 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-03-30T13:21:54.705 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T13:21:54.705 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T13:21:54.705 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T13:21:54.705 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T13:21:54.719 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:21:54.719 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-03-30T13:21:54.719 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 6 2024-03-30T13:21:54.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:54.844+0000 7f5e84bd3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:54.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:54.860+0000 7f5e84bd3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:55.043 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:21:55.040+0000 7fbb38f08640 -1 mon.a@0(leader).osd e721 definitely_dead 0 2024-03-30T13:21:55.528 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.6. 2024-03-30T13:21:55.557 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-03-30T13:21:55.558 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:21:55.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:55.668+0000 7f721ba5a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:55.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:55.680+0000 7f721ba5a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:56.517 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:21:56.548 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:21:56.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:56.660+0000 7f164e29f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:56.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:56.676+0000 7f164e29f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:57.548 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:21:57.576 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:21:57.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:57.688+0000 7f2520e1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:57.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:57.704+0000 7f2520e1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:58.551 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:21:58.580 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:21:58.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:58.700+0000 7f6634a78640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:58.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:58.716+0000 7f6634a78640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:59.567 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:21:59.593 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:21:59.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:59.712+0000 7f67db6a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:21:59.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:21:59.728+0000 7f67db6a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:00.571 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:22:00.601 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:22:00.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:00.720+0000 7f0b7e8a0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:00.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:00.740+0000 7f0b7e8a0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:01.603 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:22:01.629 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:22:01.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:01.748+0000 7f09b87f4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:01.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:01.764+0000 7f09b87f4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:02.615 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:22:02.638 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:22:02.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:02.756+0000 7fd3fd8d3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:02.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:02.772+0000 7fd3fd8d3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:03.614 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:22:03.639 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:22:03.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:03.756+0000 7f75575a5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:03.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:03.772+0000 7f75575a5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:04.621 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:22:04.649 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:22:04.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:04.768+0000 7f465d10a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:04.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:04.784+0000 7f465d10a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:05.641 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:22:05.666 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:22:05.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:05.788+0000 7fac5fa75640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:05.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:05.804+0000 7fac5fa75640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:06.660 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:22:06.685 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:22:06.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:06.800+0000 7fdc4afce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:06.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:06.820+0000 7fdc4afce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:07.667 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:22:07.695 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:22:07.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:07.812+0000 7ff0cec11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:07.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:07.832+0000 7ff0cec11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:08.681 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:22:08.711 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:22:08.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:08.832+0000 7f140e93c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:08.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:08.848+0000 7f140e93c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:22:09.728 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:22:09.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:09.848+0000 7fb840d37640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:09.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:09.864+0000 7fb840d37640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:10.724 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:22:10.750 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:22:10.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:10.868+0000 7f7a918ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:10.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:10.884+0000 7f7a918ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:11.725 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:22:11.749 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:22:11.749 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:22:11.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:11.864+0000 7fc1b0ccc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:11.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:11.876+0000 7fc1b0ccc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:12.049 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:22:12.049 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:22:12.066 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1530,"stamp":"2024-03-30T13:22:11.723842+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":238202410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":466000,"kb_used_data":164564,"kb_used_omap":67,"kb_used_meta":301244,"kb_avail":754508720,"statfs":{"total":773094113280,"available":772616929280,"internally_reserved":0,"allocated":168513536,"data_stored":129146032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":69141,"internal_metadata":308474347},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.052521"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1625,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-03-30T13:22:09.861818+0000","last_change":"2024-03-30T13:21:56.538529+0000","last_active":"2024-03-30T13:22:09.861818+0000","last_peered":"2024-03-30T13:22:09.861818+0000","last_clean":"2024-03-30T13:22:09.861818+0000","last_became_active":"2024-03-30T13:21:56.537232+0000","last_became_peered":"2024-03-30T13:21:56.537232+0000","last_unstale":"2024-03-30T13:22:09.861818+0000","last_undegraded":"2024-03-30T13:22:09.861818+0000","last_fullsized":"2024-03-30T13:22:09.861818+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:21:28.696384+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:21:28.696384+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:55.460477+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1548,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-03-30T13:22:07.112824+0000","last_change":"2024-03-30T13:21:17.333015+0000","last_active":"2024-03-30T13:22:07.112824+0000","last_peered":"2024-03-30T13:22:07.112824+0000","last_clean":"2024-03-30T13:22:07.112824+0000","last_became_active":"2024-03-30T13:21:05.571151+0000","last_became_peered":"2024-03-30T13:21:05.571151+0000","last_unstale":"2024-03-30T13:22:07.112824+0000","last_undegraded":"2024-03-30T13:22:07.112824+0000","last_fullsized":"2024-03-30T13:22:07.112824+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:17.332777+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:21:17.332777+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:44.049961+0000","scrub_duration":695,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":318,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-03-30T13:22:07.121811+0000","last_change":"2024-03-30T13:22:07.121892+0000","last_active":"2024-03-30T13:22:07.121811+0000","last_peered":"2024-03-30T13:22:07.121811+0000","last_clean":"2024-03-30T13:22:07.121811+0000","last_became_active":"2024-03-30T13:20:14.422234+0000","last_became_peered":"2024-03-30T13:20:14.422234+0000","last_unstale":"2024-03-30T13:22:07.121811+0000","last_undegraded":"2024-03-30T13:22:07.121811+0000","last_fullsized":"2024-03-30T13:22:07.121811+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:22:07.121601+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:25.085038+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":323,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-03-30T13:22:07.109320+0000","last_change":"2024-03-30T13:21:56.542316+0000","last_active":"2024-03-30T13:22:07.109320+0000","last_peered":"2024-03-30T13:22:07.109320+0000","last_clean":"2024-03-30T13:22:07.109320+0000","last_became_active":"2024-03-30T13:21:56.541153+0000","last_became_peered":"2024-03-30T13:21:56.541153+0000","last_unstale":"2024-03-30T13:22:07.109320+0000","last_undegraded":"2024-03-30T13:22:07.109320+0000","last_fullsized":"2024-03-30T13:22:07.109320+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:21:26.828701+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:21:26.828701+0000","last_clean_scrub_stamp":"2024-03-30T13:21:26.828701+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:26.950133+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":502,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-03-30T13:22:06.973972+0000","last_change":"2024-03-30T13:21:56.308230+0000","last_active":"2024-03-30T13:22:06.973972+0000","last_peered":"2024-03-30T13:22:06.973972+0000","last_clean":"2024-03-30T13:22:06.973972+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:22:06.973972+0000","last_undegraded":"2024-03-30T13:22:06.973972+0000","last_fullsized":"2024-03-30T13:22:06.973972+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:56.307946+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:21:56.307946+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:22.988661+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1627,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-30T13:22:09.362152+0000","last_change":"2024-03-30T13:21:21.676616+0000","last_active":"2024-03-30T13:22:09.362152+0000","last_peered":"2024-03-30T13:22:09.362152+0000","last_clean":"2024-03-30T13:22:09.362152+0000","last_became_active":"2024-03-30T13:21:05.574411+0000","last_became_peered":"2024-03-30T13:21:05.574411+0000","last_unstale":"2024-03-30T13:22:09.362152+0000","last_undegraded":"2024-03-30T13:22:09.362152+0000","last_fullsized":"2024-03-30T13:22:09.362152+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:21:21.676444+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:29.360053+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":366,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-03-30T13:22:07.109369+0000","last_change":"2024-03-30T13:21:57.102491+0000","last_active":"2024-03-30T13:22:07.109369+0000","last_peered":"2024-03-30T13:22:07.109369+0000","last_clean":"2024-03-30T13:22:07.109369+0000","last_became_active":"2024-03-30T13:20:14.421760+0000","last_became_peered":"2024-03-30T13:20:14.421760+0000","last_unstale":"2024-03-30T13:22:07.109369+0000","last_undegraded":"2024-03-30T13:22:07.109369+0000","last_fullsized":"2024-03-30T13:22:07.109369+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:21:57.102229+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:21:57.102229+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:21.918752+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":942,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-03-30T13:22:09.704814+0000","last_change":"2024-03-30T13:21:56.541906+0000","last_active":"2024-03-30T13:22:09.704814+0000","last_peered":"2024-03-30T13:22:09.704814+0000","last_clean":"2024-03-30T13:22:09.704814+0000","last_became_active":"2024-03-30T13:21:56.540619+0000","last_became_peered":"2024-03-30T13:21:56.540619+0000","last_unstale":"2024-03-30T13:22:09.704814+0000","last_undegraded":"2024-03-30T13:22:09.704814+0000","last_fullsized":"2024-03-30T13:22:09.704814+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:21:30.546139+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:21:30.546139+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:58.351721+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":73445600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1758,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-30T13:22:09.361956+0000","last_change":"2024-03-30T13:21:05.574623+0000","last_active":"2024-03-30T13:22:09.361956+0000","last_peered":"2024-03-30T13:22:09.361956+0000","last_clean":"2024-03-30T13:22:09.361956+0000","last_became_active":"2024-03-30T13:21:05.574131+0000","last_became_peered":"2024-03-30T13:21:05.574131+0000","last_unstale":"2024-03-30T13:22:09.361956+0000","last_undegraded":"2024-03-30T13:22:09.361956+0000","last_fullsized":"2024-03-30T13:22:09.361956+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:56.407671+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:20:56.407671+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:11.138773+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1393,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-03-30T13:22:07.112698+0000","last_change":"2024-03-30T13:21:57.711005+0000","last_active":"2024-03-30T13:22:07.112698+0000","last_peered":"2024-03-30T13:22:07.112698+0000","last_clean":"2024-03-30T13:22:07.112698+0000","last_became_active":"2024-03-30T13:21:56.531330+0000","last_became_peered":"2024-03-30T13:21:56.531330+0000","last_unstale":"2024-03-30T13:22:07.112698+0000","last_undegraded":"2024-03-30T13:22:07.112698+0000","last_fullsized":"2024-03-30T13:22:07.112698+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:57.710868+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:21:57.710868+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:10.029079+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1652,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-03-30T13:22:09.861985+0000","last_change":"2024-03-30T13:21:57.935968+0000","last_active":"2024-03-30T13:22:09.861985+0000","last_peered":"2024-03-30T13:22:09.861985+0000","last_clean":"2024-03-30T13:22:09.861985+0000","last_became_active":"2024-03-30T13:21:56.537406+0000","last_became_peered":"2024-03-30T13:21:56.537406+0000","last_unstale":"2024-03-30T13:22:09.861985+0000","last_undegraded":"2024-03-30T13:22:09.861985+0000","last_fullsized":"2024-03-30T13:22:09.861985+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:21:57.935530+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:21:57.935530+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:14.327616+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1710,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-03-30T13:22:09.861513+0000","last_change":"2024-03-30T13:21:56.543824+0000","last_active":"2024-03-30T13:22:09.861513+0000","last_peered":"2024-03-30T13:22:09.861513+0000","last_clean":"2024-03-30T13:22:09.861513+0000","last_became_active":"2024-03-30T13:21:56.542214+0000","last_became_peered":"2024-03-30T13:21:56.542214+0000","last_unstale":"2024-03-30T13:22:09.861513+0000","last_undegraded":"2024-03-30T13:22:09.861513+0000","last_fullsized":"2024-03-30T13:22:09.861513+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:21:49.068011+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:21:49.068011+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:13.696559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":536,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-03-30T13:22:06.973944+0000","last_change":"2024-03-30T13:21:33.418242+0000","last_active":"2024-03-30T13:22:06.973944+0000","last_peered":"2024-03-30T13:22:06.973944+0000","last_clean":"2024-03-30T13:22:06.973944+0000","last_became_active":"2024-03-30T13:21:05.572577+0000","last_became_peered":"2024-03-30T13:21:05.572577+0000","last_unstale":"2024-03-30T13:22:06.973944+0000","last_undegraded":"2024-03-30T13:22:06.973944+0000","last_fullsized":"2024-03-30T13:22:06.973944+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:21:33.417986+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:21:33.417986+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:41.633765+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":312,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-03-30T13:22:07.109358+0000","last_change":"2024-03-30T13:21:06.773969+0000","last_active":"2024-03-30T13:22:07.109358+0000","last_peered":"2024-03-30T13:22:07.109358+0000","last_clean":"2024-03-30T13:22:07.109358+0000","last_became_active":"2024-03-30T13:21:05.570131+0000","last_became_peered":"2024-03-30T13:21:05.570131+0000","last_unstale":"2024-03-30T13:22:07.109358+0000","last_undegraded":"2024-03-30T13:22:07.109358+0000","last_fullsized":"2024-03-30T13:22:07.109358+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:06.773612+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:21:06.773612+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:19.252741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":534,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-03-30T13:22:06.973981+0000","last_change":"2024-03-30T13:21:56.535977+0000","last_active":"2024-03-30T13:22:06.973981+0000","last_peered":"2024-03-30T13:22:06.973981+0000","last_clean":"2024-03-30T13:22:06.973981+0000","last_became_active":"2024-03-30T13:21:56.534725+0000","last_became_peered":"2024-03-30T13:21:56.534725+0000","last_unstale":"2024-03-30T13:22:06.973981+0000","last_undegraded":"2024-03-30T13:22:06.973981+0000","last_fullsized":"2024-03-30T13:22:06.973981+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:20:54.701569+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:20:54.701569+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:15.050127+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":884,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-03-30T13:22:09.704890+0000","last_change":"2024-03-30T13:21:59.583334+0000","last_active":"2024-03-30T13:22:09.704890+0000","last_peered":"2024-03-30T13:22:09.704890+0000","last_clean":"2024-03-30T13:22:09.704890+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:22:09.704890+0000","last_undegraded":"2024-03-30T13:22:09.704890+0000","last_fullsized":"2024-03-30T13:22:09.704890+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:21:59.583187+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:21:59.583187+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:19.245095+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":238202410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387717,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61092,"kb_used_data":23892,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94310748,"statfs":{"total":96636764160,"available":96574205952,"internally_reserved":0,"allocated":24465408,"data_stored":19530868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157532,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66248,"kb_used_data":28600,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94305592,"statfs":{"total":96636764160,"available":96568926208,"internally_reserved":0,"allocated":29286400,"data_stored":24372263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":677,"seq":2907692859417,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62112,"kb_used_data":20304,"kb_used_omap":13,"kb_used_meta":41778,"kb_avail":94309728,"statfs":{"total":96636764160,"available":96573161472,"internally_reserved":0,"allocated":20791296,"data_stored":15877159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14287,"internal_metadata":42780721},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509763,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50436,"kb_used_data":15096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94321404,"statfs":{"total":96636764160,"available":96585117696,"internally_reserved":0,"allocated":15458304,"data_stored":10530565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375999042,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51572,"kb_used_data":14372,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94320268,"statfs":{"total":96636764160,"available":96583954432,"internally_reserved":0,"allocated":14716928,"data_stored":9789189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509764,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57612,"kb_used_data":22272,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94314228,"statfs":{"total":96636764160,"available":96577769472,"internally_reserved":0,"allocated":22806528,"data_stored":17878789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742436,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59376,"kb_used_data":18780,"kb_used_omap":15,"kb_used_meta":40560,"kb_avail":94312464,"statfs":{"total":96636764160,"available":96575963136,"internally_reserved":0,"allocated":19230720,"data_stored":14323384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":41533891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":699,"seq":3002182139919,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57552,"kb_used_data":21248,"kb_used_omap":5,"kb_used_meta":36282,"kb_avail":94314288,"statfs":{"total":96636764160,"available":96577830912,"internally_reserved":0,"allocated":21757952,"data_stored":16843815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":37152866},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:22:12.067 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:22:12.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:12.172+0000 7fa426103640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:12.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:12.184+0000 7fa426103640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:12.353 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:22:12.353 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:22:12.370 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1531,"stamp":"2024-03-30T13:22:12.246060+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":238202410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":466000,"kb_used_data":164564,"kb_used_omap":67,"kb_used_meta":301244,"kb_avail":754508720,"statfs":{"total":773094113280,"available":772616929280,"internally_reserved":0,"allocated":168513536,"data_stored":129146032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":69141,"internal_metadata":308474347},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001084"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1625,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-03-30T13:22:09.861818+0000","last_change":"2024-03-30T13:21:56.538529+0000","last_active":"2024-03-30T13:22:09.861818+0000","last_peered":"2024-03-30T13:22:09.861818+0000","last_clean":"2024-03-30T13:22:09.861818+0000","last_became_active":"2024-03-30T13:21:56.537232+0000","last_became_peered":"2024-03-30T13:21:56.537232+0000","last_unstale":"2024-03-30T13:22:09.861818+0000","last_undegraded":"2024-03-30T13:22:09.861818+0000","last_fullsized":"2024-03-30T13:22:09.861818+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:21:28.696384+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:21:28.696384+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:55.460477+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1548,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-03-30T13:22:07.112824+0000","last_change":"2024-03-30T13:21:17.333015+0000","last_active":"2024-03-30T13:22:07.112824+0000","last_peered":"2024-03-30T13:22:07.112824+0000","last_clean":"2024-03-30T13:22:07.112824+0000","last_became_active":"2024-03-30T13:21:05.571151+0000","last_became_peered":"2024-03-30T13:21:05.571151+0000","last_unstale":"2024-03-30T13:22:07.112824+0000","last_undegraded":"2024-03-30T13:22:07.112824+0000","last_fullsized":"2024-03-30T13:22:07.112824+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:17.332777+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:21:17.332777+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:44.049961+0000","scrub_duration":695,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":318,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-03-30T13:22:07.121811+0000","last_change":"2024-03-30T13:22:07.121892+0000","last_active":"2024-03-30T13:22:07.121811+0000","last_peered":"2024-03-30T13:22:07.121811+0000","last_clean":"2024-03-30T13:22:07.121811+0000","last_became_active":"2024-03-30T13:20:14.422234+0000","last_became_peered":"2024-03-30T13:20:14.422234+0000","last_unstale":"2024-03-30T13:22:07.121811+0000","last_undegraded":"2024-03-30T13:22:07.121811+0000","last_fullsized":"2024-03-30T13:22:07.121811+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:22:07.121601+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:25.085038+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":323,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-03-30T13:22:07.109320+0000","last_change":"2024-03-30T13:21:56.542316+0000","last_active":"2024-03-30T13:22:07.109320+0000","last_peered":"2024-03-30T13:22:07.109320+0000","last_clean":"2024-03-30T13:22:07.109320+0000","last_became_active":"2024-03-30T13:21:56.541153+0000","last_became_peered":"2024-03-30T13:21:56.541153+0000","last_unstale":"2024-03-30T13:22:07.109320+0000","last_undegraded":"2024-03-30T13:22:07.109320+0000","last_fullsized":"2024-03-30T13:22:07.109320+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:21:26.828701+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:21:26.828701+0000","last_clean_scrub_stamp":"2024-03-30T13:21:26.828701+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:26.950133+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":502,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-03-30T13:22:06.973972+0000","last_change":"2024-03-30T13:21:56.308230+0000","last_active":"2024-03-30T13:22:06.973972+0000","last_peered":"2024-03-30T13:22:06.973972+0000","last_clean":"2024-03-30T13:22:06.973972+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:22:06.973972+0000","last_undegraded":"2024-03-30T13:22:06.973972+0000","last_fullsized":"2024-03-30T13:22:06.973972+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:56.307946+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:21:56.307946+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:22.988661+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1627,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-30T13:22:09.362152+0000","last_change":"2024-03-30T13:21:21.676616+0000","last_active":"2024-03-30T13:22:09.362152+0000","last_peered":"2024-03-30T13:22:09.362152+0000","last_clean":"2024-03-30T13:22:09.362152+0000","last_became_active":"2024-03-30T13:21:05.574411+0000","last_became_peered":"2024-03-30T13:21:05.574411+0000","last_unstale":"2024-03-30T13:22:09.362152+0000","last_undegraded":"2024-03-30T13:22:09.362152+0000","last_fullsized":"2024-03-30T13:22:09.362152+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:21:21.676444+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:29.360053+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":366,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-03-30T13:22:07.109369+0000","last_change":"2024-03-30T13:21:57.102491+0000","last_active":"2024-03-30T13:22:07.109369+0000","last_peered":"2024-03-30T13:22:07.109369+0000","last_clean":"2024-03-30T13:22:07.109369+0000","last_became_active":"2024-03-30T13:20:14.421760+0000","last_became_peered":"2024-03-30T13:20:14.421760+0000","last_unstale":"2024-03-30T13:22:07.109369+0000","last_undegraded":"2024-03-30T13:22:07.109369+0000","last_fullsized":"2024-03-30T13:22:07.109369+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:21:57.102229+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:21:57.102229+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:21.918752+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":942,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-03-30T13:22:09.704814+0000","last_change":"2024-03-30T13:21:56.541906+0000","last_active":"2024-03-30T13:22:09.704814+0000","last_peered":"2024-03-30T13:22:09.704814+0000","last_clean":"2024-03-30T13:22:09.704814+0000","last_became_active":"2024-03-30T13:21:56.540619+0000","last_became_peered":"2024-03-30T13:21:56.540619+0000","last_unstale":"2024-03-30T13:22:09.704814+0000","last_undegraded":"2024-03-30T13:22:09.704814+0000","last_fullsized":"2024-03-30T13:22:09.704814+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:21:30.546139+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:21:30.546139+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:58.351721+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":73445600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1758,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-30T13:22:09.361956+0000","last_change":"2024-03-30T13:21:05.574623+0000","last_active":"2024-03-30T13:22:09.361956+0000","last_peered":"2024-03-30T13:22:09.361956+0000","last_clean":"2024-03-30T13:22:09.361956+0000","last_became_active":"2024-03-30T13:21:05.574131+0000","last_became_peered":"2024-03-30T13:21:05.574131+0000","last_unstale":"2024-03-30T13:22:09.361956+0000","last_undegraded":"2024-03-30T13:22:09.361956+0000","last_fullsized":"2024-03-30T13:22:09.361956+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:56.407671+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:20:56.407671+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:11.138773+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1393,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-03-30T13:22:07.112698+0000","last_change":"2024-03-30T13:21:57.711005+0000","last_active":"2024-03-30T13:22:07.112698+0000","last_peered":"2024-03-30T13:22:07.112698+0000","last_clean":"2024-03-30T13:22:07.112698+0000","last_became_active":"2024-03-30T13:21:56.531330+0000","last_became_peered":"2024-03-30T13:21:56.531330+0000","last_unstale":"2024-03-30T13:22:07.112698+0000","last_undegraded":"2024-03-30T13:22:07.112698+0000","last_fullsized":"2024-03-30T13:22:07.112698+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:57.710868+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:21:57.710868+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:10.029079+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1652,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-03-30T13:22:09.861985+0000","last_change":"2024-03-30T13:21:57.935968+0000","last_active":"2024-03-30T13:22:09.861985+0000","last_peered":"2024-03-30T13:22:09.861985+0000","last_clean":"2024-03-30T13:22:09.861985+0000","last_became_active":"2024-03-30T13:21:56.537406+0000","last_became_peered":"2024-03-30T13:21:56.537406+0000","last_unstale":"2024-03-30T13:22:09.861985+0000","last_undegraded":"2024-03-30T13:22:09.861985+0000","last_fullsized":"2024-03-30T13:22:09.861985+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:21:57.935530+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:21:57.935530+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:14.327616+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1710,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-03-30T13:22:09.861513+0000","last_change":"2024-03-30T13:21:56.543824+0000","last_active":"2024-03-30T13:22:09.861513+0000","last_peered":"2024-03-30T13:22:09.861513+0000","last_clean":"2024-03-30T13:22:09.861513+0000","last_became_active":"2024-03-30T13:21:56.542214+0000","last_became_peered":"2024-03-30T13:21:56.542214+0000","last_unstale":"2024-03-30T13:22:09.861513+0000","last_undegraded":"2024-03-30T13:22:09.861513+0000","last_fullsized":"2024-03-30T13:22:09.861513+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:21:49.068011+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:21:49.068011+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:13.696559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":536,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-03-30T13:22:06.973944+0000","last_change":"2024-03-30T13:21:33.418242+0000","last_active":"2024-03-30T13:22:06.973944+0000","last_peered":"2024-03-30T13:22:06.973944+0000","last_clean":"2024-03-30T13:22:06.973944+0000","last_became_active":"2024-03-30T13:21:05.572577+0000","last_became_peered":"2024-03-30T13:21:05.572577+0000","last_unstale":"2024-03-30T13:22:06.973944+0000","last_undegraded":"2024-03-30T13:22:06.973944+0000","last_fullsized":"2024-03-30T13:22:06.973944+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:21:33.417986+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:21:33.417986+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:41.633765+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":312,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-03-30T13:22:07.109358+0000","last_change":"2024-03-30T13:21:06.773969+0000","last_active":"2024-03-30T13:22:07.109358+0000","last_peered":"2024-03-30T13:22:07.109358+0000","last_clean":"2024-03-30T13:22:07.109358+0000","last_became_active":"2024-03-30T13:21:05.570131+0000","last_became_peered":"2024-03-30T13:21:05.570131+0000","last_unstale":"2024-03-30T13:22:07.109358+0000","last_undegraded":"2024-03-30T13:22:07.109358+0000","last_fullsized":"2024-03-30T13:22:07.109358+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:06.773612+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:21:06.773612+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:19.252741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":534,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-03-30T13:22:06.973981+0000","last_change":"2024-03-30T13:21:56.535977+0000","last_active":"2024-03-30T13:22:06.973981+0000","last_peered":"2024-03-30T13:22:06.973981+0000","last_clean":"2024-03-30T13:22:06.973981+0000","last_became_active":"2024-03-30T13:21:56.534725+0000","last_became_peered":"2024-03-30T13:21:56.534725+0000","last_unstale":"2024-03-30T13:22:06.973981+0000","last_undegraded":"2024-03-30T13:22:06.973981+0000","last_fullsized":"2024-03-30T13:22:06.973981+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:20:54.701569+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:20:54.701569+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:15.050127+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":884,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-03-30T13:22:09.704890+0000","last_change":"2024-03-30T13:21:59.583334+0000","last_active":"2024-03-30T13:22:09.704890+0000","last_peered":"2024-03-30T13:22:09.704890+0000","last_clean":"2024-03-30T13:22:09.704890+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:22:09.704890+0000","last_undegraded":"2024-03-30T13:22:09.704890+0000","last_fullsized":"2024-03-30T13:22:09.704890+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:21:59.583187+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:21:59.583187+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:19.245095+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":238202410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387717,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61092,"kb_used_data":23892,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94310748,"statfs":{"total":96636764160,"available":96574205952,"internally_reserved":0,"allocated":24465408,"data_stored":19530868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157532,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66248,"kb_used_data":28600,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94305592,"statfs":{"total":96636764160,"available":96568926208,"internally_reserved":0,"allocated":29286400,"data_stored":24372263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":677,"seq":2907692859417,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62112,"kb_used_data":20304,"kb_used_omap":13,"kb_used_meta":41778,"kb_avail":94309728,"statfs":{"total":96636764160,"available":96573161472,"internally_reserved":0,"allocated":20791296,"data_stored":15877159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14287,"internal_metadata":42780721},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509763,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50436,"kb_used_data":15096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94321404,"statfs":{"total":96636764160,"available":96585117696,"internally_reserved":0,"allocated":15458304,"data_stored":10530565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375999042,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51572,"kb_used_data":14372,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94320268,"statfs":{"total":96636764160,"available":96583954432,"internally_reserved":0,"allocated":14716928,"data_stored":9789189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509764,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57612,"kb_used_data":22272,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94314228,"statfs":{"total":96636764160,"available":96577769472,"internally_reserved":0,"allocated":22806528,"data_stored":17878789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742436,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59376,"kb_used_data":18780,"kb_used_omap":15,"kb_used_meta":40560,"kb_avail":94312464,"statfs":{"total":96636764160,"available":96575963136,"internally_reserved":0,"allocated":19230720,"data_stored":14323384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":41533891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":699,"seq":3002182139919,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57552,"kb_used_data":21248,"kb_used_omap":5,"kb_used_meta":36282,"kb_avail":94314288,"statfs":{"total":96636764160,"available":96577830912,"internally_reserved":0,"allocated":21757952,"data_stored":16843815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":37152866},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:22:12.371 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:22:12.371 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:22:12.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:12.472+0000 7f58481c7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:12.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:12.484+0000 7f58481c7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:12.651 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:22:12.651 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":738,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:22:11.720178+0000","last_up_change":"2024-03-30T13:21:55.513998+0000","last_in_change":"2024-03-30T13:21:55.513998+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":36,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":716,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":461,"last_clean_end":575,"up_from":581,"up_thru":722,"down_at":576,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6818","nonce":4030404750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6821","nonce":4030404750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6828","nonce":4030404750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6825","nonce":4030404750}]},"public_addr":"172.21.15.67:6818/4030404750","cluster_addr":"172.21.15.67:6821/4030404750","heartbeat_back_addr":"172.21.15.67:6828/4030404750","heartbeat_front_addr":"172.21.15.67:6825/4030404750","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":668,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":393,"last_clean_end":693,"up_from":699,"up_thru":717,"down_at":694,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1679277090},{"type":"v1","addr":"172.21.15.67:6801","nonce":1679277090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1679277090},{"type":"v1","addr":"172.21.15.67:6803","nonce":1679277090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1679277090},{"type":"v1","addr":"172.21.15.67:6807","nonce":1679277090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1679277090},{"type":"v1","addr":"172.21.15.67:6805","nonce":1679277090}]},"public_addr":"172.21.15.67:6801/1679277090","cluster_addr":"172.21.15.67:6803/1679277090","heartbeat_back_addr":"172.21.15.67:6807/1679277090","heartbeat_front_addr":"172.21.15.67:6805/1679277090","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":559,"last_clean_end":666,"up_from":677,"up_thru":722,"down_at":667,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1606011629},{"type":"v1","addr":"172.21.15.82:6803","nonce":1606011629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":1606011629},{"type":"v1","addr":"172.21.15.82:6807","nonce":1606011629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":1606011629},{"type":"v1","addr":"172.21.15.82:6815","nonce":1606011629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":1606011629},{"type":"v1","addr":"172.21.15.82:6811","nonce":1606011629}]},"public_addr":"172.21.15.82:6803/1606011629","cluster_addr":"172.21.15.82:6807/1606011629","heartbeat_back_addr":"172.21.15.82:6815/1606011629","heartbeat_front_addr":"172.21.15.82:6811/1606011629","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":488,"last_clean_end":506,"up_from":515,"up_thru":722,"down_at":507,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6817","nonce":292505453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6819","nonce":292505453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6823","nonce":292505453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6821","nonce":292505453}]},"public_addr":"172.21.15.82:6817/292505453","cluster_addr":"172.21.15.82:6819/292505453","heartbeat_back_addr":"172.21.15.82:6823/292505453","heartbeat_front_addr":"172.21.15.82:6821/292505453","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":629,"last_clean_end":715,"up_from":722,"up_thru":722,"down_at":716,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6801","nonce":326680821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6806","nonce":326680821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6814","nonce":326680821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6810","nonce":326680821}]},"public_addr":"172.21.15.82:6801/326680821","cluster_addr":"172.21.15.82:6806/326680821","heartbeat_back_addr":"172.21.15.82:6814/326680821","heartbeat_front_addr":"172.21.15.82:6810/326680821","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":604,"last_clean_end":645,"up_from":650,"up_thru":722,"down_at":646,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6825","nonce":1497758647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6827","nonce":1497758647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6831","nonce":1497758647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6829","nonce":1497758647}]},"public_addr":"172.21.15.82:6825/1497758647","cluster_addr":"172.21.15.82:6827/1497758647","heartbeat_back_addr":"172.21.15.82:6831/1497758647","heartbeat_front_addr":"172.21.15.82:6829/1497758647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:21:22.724073+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:21:23.986892+0000","dead_epoch":576},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:20:59.842948+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:21:06.735033+0000","dead_epoch":694},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:21:44.551643+0000","dead_epoch":667},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:21:14.902871+0000","dead_epoch":507},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:21:08.787068+0000","dead_epoch":716},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:21:39.718742+0000","dead_epoch":646}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:22:12.668 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:22:12.669 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:22:12.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:12.772+0000 7f5bcaf9e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:12.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:12.788+0000 7f5bcaf9e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:12.957 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:22:12.976 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:22:13.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:13.092+0000 7fd748b5d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:13.108+0000 7fd748b5d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:13.282 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:22:13.282 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:22:13.299 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:22:13.299 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:22:13.299 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:22:13.300 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:22:13.300 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 5, 1, 7, 4, 3, 6] 2024-03-30T13:22:13.300 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:22:13.300 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:22:13.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:13.404+0000 7ffaae1d5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:13.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:13.420+0000 7ffaae1d5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:13.593 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:22:13.593 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:22:13.611 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1531,"stamp":"2024-03-30T13:22:12.246060+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":238202410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":466000,"kb_used_data":164564,"kb_used_omap":67,"kb_used_meta":301244,"kb_avail":754508720,"statfs":{"total":773094113280,"available":772616929280,"internally_reserved":0,"allocated":168513536,"data_stored":129146032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":69141,"internal_metadata":308474347},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001084"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1625,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-03-30T13:22:09.861818+0000","last_change":"2024-03-30T13:21:56.538529+0000","last_active":"2024-03-30T13:22:09.861818+0000","last_peered":"2024-03-30T13:22:09.861818+0000","last_clean":"2024-03-30T13:22:09.861818+0000","last_became_active":"2024-03-30T13:21:56.537232+0000","last_became_peered":"2024-03-30T13:21:56.537232+0000","last_unstale":"2024-03-30T13:22:09.861818+0000","last_undegraded":"2024-03-30T13:22:09.861818+0000","last_fullsized":"2024-03-30T13:22:09.861818+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:21:28.696384+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:21:28.696384+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:55.460477+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1548,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-03-30T13:22:07.112824+0000","last_change":"2024-03-30T13:21:17.333015+0000","last_active":"2024-03-30T13:22:07.112824+0000","last_peered":"2024-03-30T13:22:07.112824+0000","last_clean":"2024-03-30T13:22:07.112824+0000","last_became_active":"2024-03-30T13:21:05.571151+0000","last_became_peered":"2024-03-30T13:21:05.571151+0000","last_unstale":"2024-03-30T13:22:07.112824+0000","last_undegraded":"2024-03-30T13:22:07.112824+0000","last_fullsized":"2024-03-30T13:22:07.112824+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:17.332777+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:21:17.332777+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:44.049961+0000","scrub_duration":695,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":318,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-03-30T13:22:07.121811+0000","last_change":"2024-03-30T13:22:07.121892+0000","last_active":"2024-03-30T13:22:07.121811+0000","last_peered":"2024-03-30T13:22:07.121811+0000","last_clean":"2024-03-30T13:22:07.121811+0000","last_became_active":"2024-03-30T13:20:14.422234+0000","last_became_peered":"2024-03-30T13:20:14.422234+0000","last_unstale":"2024-03-30T13:22:07.121811+0000","last_undegraded":"2024-03-30T13:22:07.121811+0000","last_fullsized":"2024-03-30T13:22:07.121811+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:22:07.121601+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:25.085038+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":323,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-03-30T13:22:07.109320+0000","last_change":"2024-03-30T13:21:56.542316+0000","last_active":"2024-03-30T13:22:07.109320+0000","last_peered":"2024-03-30T13:22:07.109320+0000","last_clean":"2024-03-30T13:22:07.109320+0000","last_became_active":"2024-03-30T13:21:56.541153+0000","last_became_peered":"2024-03-30T13:21:56.541153+0000","last_unstale":"2024-03-30T13:22:07.109320+0000","last_undegraded":"2024-03-30T13:22:07.109320+0000","last_fullsized":"2024-03-30T13:22:07.109320+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:21:26.828701+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:21:26.828701+0000","last_clean_scrub_stamp":"2024-03-30T13:21:26.828701+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:26.950133+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":502,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-03-30T13:22:06.973972+0000","last_change":"2024-03-30T13:21:56.308230+0000","last_active":"2024-03-30T13:22:06.973972+0000","last_peered":"2024-03-30T13:22:06.973972+0000","last_clean":"2024-03-30T13:22:06.973972+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:22:06.973972+0000","last_undegraded":"2024-03-30T13:22:06.973972+0000","last_fullsized":"2024-03-30T13:22:06.973972+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:56.307946+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:21:56.307946+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:22.988661+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1627,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-30T13:22:09.362152+0000","last_change":"2024-03-30T13:21:21.676616+0000","last_active":"2024-03-30T13:22:09.362152+0000","last_peered":"2024-03-30T13:22:09.362152+0000","last_clean":"2024-03-30T13:22:09.362152+0000","last_became_active":"2024-03-30T13:21:05.574411+0000","last_became_peered":"2024-03-30T13:21:05.574411+0000","last_unstale":"2024-03-30T13:22:09.362152+0000","last_undegraded":"2024-03-30T13:22:09.362152+0000","last_fullsized":"2024-03-30T13:22:09.362152+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:21:21.676444+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:29.360053+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":366,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-03-30T13:22:07.109369+0000","last_change":"2024-03-30T13:21:57.102491+0000","last_active":"2024-03-30T13:22:07.109369+0000","last_peered":"2024-03-30T13:22:07.109369+0000","last_clean":"2024-03-30T13:22:07.109369+0000","last_became_active":"2024-03-30T13:20:14.421760+0000","last_became_peered":"2024-03-30T13:20:14.421760+0000","last_unstale":"2024-03-30T13:22:07.109369+0000","last_undegraded":"2024-03-30T13:22:07.109369+0000","last_fullsized":"2024-03-30T13:22:07.109369+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:21:57.102229+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:21:57.102229+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:21.918752+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":942,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-03-30T13:22:09.704814+0000","last_change":"2024-03-30T13:21:56.541906+0000","last_active":"2024-03-30T13:22:09.704814+0000","last_peered":"2024-03-30T13:22:09.704814+0000","last_clean":"2024-03-30T13:22:09.704814+0000","last_became_active":"2024-03-30T13:21:56.540619+0000","last_became_peered":"2024-03-30T13:21:56.540619+0000","last_unstale":"2024-03-30T13:22:09.704814+0000","last_undegraded":"2024-03-30T13:22:09.704814+0000","last_fullsized":"2024-03-30T13:22:09.704814+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:21:30.546139+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:21:30.546139+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:58.351721+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":73445600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1758,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-03-30T13:22:09.361956+0000","last_change":"2024-03-30T13:21:05.574623+0000","last_active":"2024-03-30T13:22:09.361956+0000","last_peered":"2024-03-30T13:22:09.361956+0000","last_clean":"2024-03-30T13:22:09.361956+0000","last_became_active":"2024-03-30T13:21:05.574131+0000","last_became_peered":"2024-03-30T13:21:05.574131+0000","last_unstale":"2024-03-30T13:22:09.361956+0000","last_undegraded":"2024-03-30T13:22:09.361956+0000","last_fullsized":"2024-03-30T13:22:09.361956+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:56.407671+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:20:56.407671+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:11.138773+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1393,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-03-30T13:22:07.112698+0000","last_change":"2024-03-30T13:21:57.711005+0000","last_active":"2024-03-30T13:22:07.112698+0000","last_peered":"2024-03-30T13:22:07.112698+0000","last_clean":"2024-03-30T13:22:07.112698+0000","last_became_active":"2024-03-30T13:21:56.531330+0000","last_became_peered":"2024-03-30T13:21:56.531330+0000","last_unstale":"2024-03-30T13:22:07.112698+0000","last_undegraded":"2024-03-30T13:22:07.112698+0000","last_fullsized":"2024-03-30T13:22:07.112698+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:57.710868+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:21:57.710868+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:10.029079+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1652,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-03-30T13:22:09.861985+0000","last_change":"2024-03-30T13:21:57.935968+0000","last_active":"2024-03-30T13:22:09.861985+0000","last_peered":"2024-03-30T13:22:09.861985+0000","last_clean":"2024-03-30T13:22:09.861985+0000","last_became_active":"2024-03-30T13:21:56.537406+0000","last_became_peered":"2024-03-30T13:21:56.537406+0000","last_unstale":"2024-03-30T13:22:09.861985+0000","last_undegraded":"2024-03-30T13:22:09.861985+0000","last_fullsized":"2024-03-30T13:22:09.861985+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:21:57.935530+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:21:57.935530+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:14.327616+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1710,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-03-30T13:22:09.861513+0000","last_change":"2024-03-30T13:21:56.543824+0000","last_active":"2024-03-30T13:22:09.861513+0000","last_peered":"2024-03-30T13:22:09.861513+0000","last_clean":"2024-03-30T13:22:09.861513+0000","last_became_active":"2024-03-30T13:21:56.542214+0000","last_became_peered":"2024-03-30T13:21:56.542214+0000","last_unstale":"2024-03-30T13:22:09.861513+0000","last_undegraded":"2024-03-30T13:22:09.861513+0000","last_fullsized":"2024-03-30T13:22:09.861513+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:21:49.068011+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:21:49.068011+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:13.696559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":536,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-03-30T13:22:06.973944+0000","last_change":"2024-03-30T13:21:33.418242+0000","last_active":"2024-03-30T13:22:06.973944+0000","last_peered":"2024-03-30T13:22:06.973944+0000","last_clean":"2024-03-30T13:22:06.973944+0000","last_became_active":"2024-03-30T13:21:05.572577+0000","last_became_peered":"2024-03-30T13:21:05.572577+0000","last_unstale":"2024-03-30T13:22:06.973944+0000","last_undegraded":"2024-03-30T13:22:06.973944+0000","last_fullsized":"2024-03-30T13:22:06.973944+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:21:33.417986+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:21:33.417986+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:41.633765+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":312,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-03-30T13:22:07.109358+0000","last_change":"2024-03-30T13:21:06.773969+0000","last_active":"2024-03-30T13:22:07.109358+0000","last_peered":"2024-03-30T13:22:07.109358+0000","last_clean":"2024-03-30T13:22:07.109358+0000","last_became_active":"2024-03-30T13:21:05.570131+0000","last_became_peered":"2024-03-30T13:21:05.570131+0000","last_unstale":"2024-03-30T13:22:07.109358+0000","last_undegraded":"2024-03-30T13:22:07.109358+0000","last_fullsized":"2024-03-30T13:22:07.109358+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:06.773612+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:21:06.773612+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:19.252741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":534,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-03-30T13:22:06.973981+0000","last_change":"2024-03-30T13:21:56.535977+0000","last_active":"2024-03-30T13:22:06.973981+0000","last_peered":"2024-03-30T13:22:06.973981+0000","last_clean":"2024-03-30T13:22:06.973981+0000","last_became_active":"2024-03-30T13:21:56.534725+0000","last_became_peered":"2024-03-30T13:21:56.534725+0000","last_unstale":"2024-03-30T13:22:06.973981+0000","last_undegraded":"2024-03-30T13:22:06.973981+0000","last_fullsized":"2024-03-30T13:22:06.973981+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:20:54.701569+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:20:54.701569+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:15.050127+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":884,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-03-30T13:22:09.704890+0000","last_change":"2024-03-30T13:21:59.583334+0000","last_active":"2024-03-30T13:22:09.704890+0000","last_peered":"2024-03-30T13:22:09.704890+0000","last_clean":"2024-03-30T13:22:09.704890+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:22:09.704890+0000","last_undegraded":"2024-03-30T13:22:09.704890+0000","last_fullsized":"2024-03-30T13:22:09.704890+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:21:59.583187+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:21:59.583187+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:19.245095+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":238202410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387717,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61092,"kb_used_data":23892,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94310748,"statfs":{"total":96636764160,"available":96574205952,"internally_reserved":0,"allocated":24465408,"data_stored":19530868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157532,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66248,"kb_used_data":28600,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94305592,"statfs":{"total":96636764160,"available":96568926208,"internally_reserved":0,"allocated":29286400,"data_stored":24372263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":677,"seq":2907692859417,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62112,"kb_used_data":20304,"kb_used_omap":13,"kb_used_meta":41778,"kb_avail":94309728,"statfs":{"total":96636764160,"available":96573161472,"internally_reserved":0,"allocated":20791296,"data_stored":15877159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14287,"internal_metadata":42780721},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509763,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50436,"kb_used_data":15096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94321404,"statfs":{"total":96636764160,"available":96585117696,"internally_reserved":0,"allocated":15458304,"data_stored":10530565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375999042,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51572,"kb_used_data":14372,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94320268,"statfs":{"total":96636764160,"available":96583954432,"internally_reserved":0,"allocated":14716928,"data_stored":9789189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509764,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57612,"kb_used_data":22272,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94314228,"statfs":{"total":96636764160,"available":96577769472,"internally_reserved":0,"allocated":22806528,"data_stored":17878789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742436,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59376,"kb_used_data":18780,"kb_used_omap":15,"kb_used_meta":40560,"kb_avail":94312464,"statfs":{"total":96636764160,"available":96575963136,"internally_reserved":0,"allocated":19230720,"data_stored":14323384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":41533891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":699,"seq":3002182139919,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57552,"kb_used_data":21248,"kb_used_omap":5,"kb_used_meta":36282,"kb_avail":94314288,"statfs":{"total":96636764160,"available":96577830912,"internally_reserved":0,"allocated":21757952,"data_stored":16843815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6046,"internal_metadata":37152866},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:22:13.612 INFO:tasks.thrashosds.thrasher:Choosing PG 1.4 with acting set [3, 5, 7] 2024-03-30T13:22:13.612 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 2, 5, 1, 7, 4, 3, 6] 2024-03-30T13:22:13.612 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-30T13:22:13.612 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:22:13.702 INFO:tasks.ceph.osd.3:Stopped 2024-03-30T13:22:13.702 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 2, 5, 1, 7, 4, 3, 6] 2024-03-30T13:22:13.702 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 3 2024-03-30T13:22:13.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:13.820+0000 7f38d65d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:13.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:13.836+0000 7f38d65d4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:14.017 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:22:14.016+0000 7fbb38f08640 -1 mon.a@0(leader).osd e739 definitely_dead 0 2024-03-30T13:22:14.760 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.3. 2024-03-30T13:22:14.784 INFO:tasks.thrashosds.thrasher:dead_osds=[3], live_osds=[0, 2, 5, 1, 7, 4, 6] 2024-03-30T13:22:14.784 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:22:14.784 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:22:14.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:14.900+0000 7f4cb6f89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:14.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:14.912+0000 7f4cb6f89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:15.089 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:22:15.089 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:22:15.110 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1534,"stamp":"2024-03-30T13:22:14.754972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":238202410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":408636,"kb_used_data":143508,"kb_used_omap":61,"kb_used_meta":264962,"kb_avail":660194244,"statfs":{"total":676457349120,"available":676038905856,"internally_reserved":0,"allocated":146952192,"data_stored":112390009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63095,"internal_metadata":271321481},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.509488"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1625,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-03-30T13:22:09.861818+0000","last_change":"2024-03-30T13:21:56.538529+0000","last_active":"2024-03-30T13:22:09.861818+0000","last_peered":"2024-03-30T13:22:09.861818+0000","last_clean":"2024-03-30T13:22:09.861818+0000","last_became_active":"2024-03-30T13:21:56.537232+0000","last_became_peered":"2024-03-30T13:21:56.537232+0000","last_unstale":"2024-03-30T13:22:09.861818+0000","last_undegraded":"2024-03-30T13:22:09.861818+0000","last_fullsized":"2024-03-30T13:22:09.861818+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:21:28.696384+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:21:28.696384+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:55.460477+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1557,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-30T13:22:12.115142+0000","last_change":"2024-03-30T13:21:17.333015+0000","last_active":"2024-03-30T13:22:12.115142+0000","last_peered":"2024-03-30T13:22:12.115142+0000","last_clean":"2024-03-30T13:22:12.115142+0000","last_became_active":"2024-03-30T13:21:05.571151+0000","last_became_peered":"2024-03-30T13:21:05.571151+0000","last_unstale":"2024-03-30T13:22:12.115142+0000","last_undegraded":"2024-03-30T13:22:12.115142+0000","last_fullsized":"2024-03-30T13:22:12.115142+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:17.332777+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:21:17.332777+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:44.049961+0000","scrub_duration":695,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":327,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-30T13:22:11.726169+0000","last_change":"2024-03-30T13:22:07.121892+0000","last_active":"2024-03-30T13:22:11.726169+0000","last_peered":"2024-03-30T13:22:11.726169+0000","last_clean":"2024-03-30T13:22:11.726169+0000","last_became_active":"2024-03-30T13:20:14.422234+0000","last_became_peered":"2024-03-30T13:20:14.422234+0000","last_unstale":"2024-03-30T13:22:11.726169+0000","last_undegraded":"2024-03-30T13:22:11.726169+0000","last_fullsized":"2024-03-30T13:22:11.726169+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:22:07.121601+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:25.085038+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":332,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-30T13:22:11.726085+0000","last_change":"2024-03-30T13:21:56.542316+0000","last_active":"2024-03-30T13:22:11.726085+0000","last_peered":"2024-03-30T13:22:11.726085+0000","last_clean":"2024-03-30T13:22:11.726085+0000","last_became_active":"2024-03-30T13:21:56.541153+0000","last_became_peered":"2024-03-30T13:21:56.541153+0000","last_unstale":"2024-03-30T13:22:11.726085+0000","last_undegraded":"2024-03-30T13:22:11.726085+0000","last_fullsized":"2024-03-30T13:22:11.726085+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:21:26.828701+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:21:26.828701+0000","last_clean_scrub_stamp":"2024-03-30T13:21:26.828701+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:26.950133+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":511,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-30T13:22:12.114800+0000","last_change":"2024-03-30T13:21:56.308230+0000","last_active":"2024-03-30T13:22:12.114800+0000","last_peered":"2024-03-30T13:22:12.114800+0000","last_clean":"2024-03-30T13:22:12.114800+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:22:12.114800+0000","last_undegraded":"2024-03-30T13:22:12.114800+0000","last_fullsized":"2024-03-30T13:22:12.114800+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:56.307946+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:21:56.307946+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:22.988661+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1627,"reported_epoch":735,"state":"stale+active+clean","last_fresh":"2024-03-30T13:22:09.362152+0000","last_change":"2024-03-30T13:21:21.676616+0000","last_active":"2024-03-30T13:22:09.362152+0000","last_peered":"2024-03-30T13:22:09.362152+0000","last_clean":"2024-03-30T13:22:09.362152+0000","last_became_active":"2024-03-30T13:21:05.574411+0000","last_became_peered":"2024-03-30T13:21:05.574411+0000","last_unstale":"2024-03-30T13:22:13.746741+0000","last_undegraded":"2024-03-30T13:22:09.362152+0000","last_fullsized":"2024-03-30T13:22:09.362152+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:21:21.676444+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:29.360053+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":375,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-30T13:22:11.726084+0000","last_change":"2024-03-30T13:21:57.102491+0000","last_active":"2024-03-30T13:22:11.726084+0000","last_peered":"2024-03-30T13:22:11.726084+0000","last_clean":"2024-03-30T13:22:11.726084+0000","last_became_active":"2024-03-30T13:20:14.421760+0000","last_became_peered":"2024-03-30T13:20:14.421760+0000","last_unstale":"2024-03-30T13:22:11.726084+0000","last_undegraded":"2024-03-30T13:22:11.726084+0000","last_fullsized":"2024-03-30T13:22:11.726084+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:21:57.102229+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:21:57.102229+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:21.918752+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":942,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-03-30T13:22:09.704814+0000","last_change":"2024-03-30T13:21:56.541906+0000","last_active":"2024-03-30T13:22:09.704814+0000","last_peered":"2024-03-30T13:22:09.704814+0000","last_clean":"2024-03-30T13:22:09.704814+0000","last_became_active":"2024-03-30T13:21:56.540619+0000","last_became_peered":"2024-03-30T13:21:56.540619+0000","last_unstale":"2024-03-30T13:22:09.704814+0000","last_undegraded":"2024-03-30T13:22:09.704814+0000","last_fullsized":"2024-03-30T13:22:09.704814+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:21:30.546139+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:21:30.546139+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:58.351721+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":73445600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1758,"reported_epoch":735,"state":"stale+active+clean","last_fresh":"2024-03-30T13:22:09.361956+0000","last_change":"2024-03-30T13:21:05.574623+0000","last_active":"2024-03-30T13:22:09.361956+0000","last_peered":"2024-03-30T13:22:09.361956+0000","last_clean":"2024-03-30T13:22:09.361956+0000","last_became_active":"2024-03-30T13:21:05.574131+0000","last_became_peered":"2024-03-30T13:21:05.574131+0000","last_unstale":"2024-03-30T13:22:13.746744+0000","last_undegraded":"2024-03-30T13:22:09.361956+0000","last_fullsized":"2024-03-30T13:22:09.361956+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:20:56.407671+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:20:56.407671+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:11.138773+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1402,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-30T13:22:12.114262+0000","last_change":"2024-03-30T13:21:57.711005+0000","last_active":"2024-03-30T13:22:12.114262+0000","last_peered":"2024-03-30T13:22:12.114262+0000","last_clean":"2024-03-30T13:22:12.114262+0000","last_became_active":"2024-03-30T13:21:56.531330+0000","last_became_peered":"2024-03-30T13:21:56.531330+0000","last_unstale":"2024-03-30T13:22:12.114262+0000","last_undegraded":"2024-03-30T13:22:12.114262+0000","last_fullsized":"2024-03-30T13:22:12.114262+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:57.710868+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:21:57.710868+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:10.029079+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1652,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-03-30T13:22:09.861985+0000","last_change":"2024-03-30T13:21:57.935968+0000","last_active":"2024-03-30T13:22:09.861985+0000","last_peered":"2024-03-30T13:22:09.861985+0000","last_clean":"2024-03-30T13:22:09.861985+0000","last_became_active":"2024-03-30T13:21:56.537406+0000","last_became_peered":"2024-03-30T13:21:56.537406+0000","last_unstale":"2024-03-30T13:22:09.861985+0000","last_undegraded":"2024-03-30T13:22:09.861985+0000","last_fullsized":"2024-03-30T13:22:09.861985+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:21:57.935530+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:21:57.935530+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:14.327616+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1710,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-03-30T13:22:09.861513+0000","last_change":"2024-03-30T13:21:56.543824+0000","last_active":"2024-03-30T13:22:09.861513+0000","last_peered":"2024-03-30T13:22:09.861513+0000","last_clean":"2024-03-30T13:22:09.861513+0000","last_became_active":"2024-03-30T13:21:56.542214+0000","last_became_peered":"2024-03-30T13:21:56.542214+0000","last_unstale":"2024-03-30T13:22:09.861513+0000","last_undegraded":"2024-03-30T13:22:09.861513+0000","last_fullsized":"2024-03-30T13:22:09.861513+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:21:49.068011+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:21:49.068011+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:13.696559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":545,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-30T13:22:12.114768+0000","last_change":"2024-03-30T13:21:33.418242+0000","last_active":"2024-03-30T13:22:12.114768+0000","last_peered":"2024-03-30T13:22:12.114768+0000","last_clean":"2024-03-30T13:22:12.114768+0000","last_became_active":"2024-03-30T13:21:05.572577+0000","last_became_peered":"2024-03-30T13:21:05.572577+0000","last_unstale":"2024-03-30T13:22:12.114768+0000","last_undegraded":"2024-03-30T13:22:12.114768+0000","last_fullsized":"2024-03-30T13:22:12.114768+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:21:33.417986+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:21:33.417986+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:41.633765+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":321,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-30T13:22:11.726061+0000","last_change":"2024-03-30T13:21:06.773969+0000","last_active":"2024-03-30T13:22:11.726061+0000","last_peered":"2024-03-30T13:22:11.726061+0000","last_clean":"2024-03-30T13:22:11.726061+0000","last_became_active":"2024-03-30T13:21:05.570131+0000","last_became_peered":"2024-03-30T13:21:05.570131+0000","last_unstale":"2024-03-30T13:22:11.726061+0000","last_undegraded":"2024-03-30T13:22:11.726061+0000","last_fullsized":"2024-03-30T13:22:11.726061+0000","mapping_epoch":699,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":700,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:06.773612+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:21:06.773612+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:19.252741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":543,"reported_epoch":738,"state":"active+clean","last_fresh":"2024-03-30T13:22:12.114838+0000","last_change":"2024-03-30T13:21:56.535977+0000","last_active":"2024-03-30T13:22:12.114838+0000","last_peered":"2024-03-30T13:22:12.114838+0000","last_clean":"2024-03-30T13:22:12.114838+0000","last_became_active":"2024-03-30T13:21:56.534725+0000","last_became_peered":"2024-03-30T13:21:56.534725+0000","last_unstale":"2024-03-30T13:22:12.114838+0000","last_undegraded":"2024-03-30T13:22:12.114838+0000","last_fullsized":"2024-03-30T13:22:12.114838+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:20:54.701569+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:20:54.701569+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:15.050127+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":884,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-03-30T13:22:09.704890+0000","last_change":"2024-03-30T13:21:59.583334+0000","last_active":"2024-03-30T13:22:09.704890+0000","last_peered":"2024-03-30T13:22:09.704890+0000","last_clean":"2024-03-30T13:22:09.704890+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:22:09.704890+0000","last_undegraded":"2024-03-30T13:22:09.704890+0000","last_fullsized":"2024-03-30T13:22:09.704890+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:21:59.583187+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:21:59.583187+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:19.245095+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":238202410,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387717,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61092,"kb_used_data":23892,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94310748,"statfs":{"total":96636764160,"available":96574205952,"internally_reserved":0,"allocated":24465408,"data_stored":19530868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157533,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66308,"kb_used_data":28660,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94305532,"statfs":{"total":96636764160,"available":96568864768,"internally_reserved":0,"allocated":29347840,"data_stored":24399698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":677,"seq":2907692859418,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62168,"kb_used_data":20364,"kb_used_omap":13,"kb_used_meta":41778,"kb_avail":94309672,"statfs":{"total":96636764160,"available":96573104128,"internally_reserved":0,"allocated":20852736,"data_stored":15904594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14287,"internal_metadata":42780721},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509763,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50436,"kb_used_data":15096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94321404,"statfs":{"total":96636764160,"available":96585117696,"internally_reserved":0,"allocated":15458304,"data_stored":10530565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375999042,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51572,"kb_used_data":14372,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94320268,"statfs":{"total":96636764160,"available":96583954432,"internally_reserved":0,"allocated":14716928,"data_stored":9789189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509764,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57612,"kb_used_data":22272,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94314228,"statfs":{"total":96636764160,"available":96577769472,"internally_reserved":0,"allocated":22806528,"data_stored":17878789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742437,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59448,"kb_used_data":18852,"kb_used_omap":15,"kb_used_meta":40560,"kb_avail":94312392,"statfs":{"total":96636764160,"available":96575889408,"internally_reserved":0,"allocated":19304448,"data_stored":14356306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":41533891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:22:15.111 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:22:40.111 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:22:40.112 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:22:40.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:40.239+0000 7fcc6e039640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:40.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:40.251+0000 7fcc6e039640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:40.425 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:22:40.425 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:22:40.443 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1551,"stamp":"2024-03-30T13:22:40.254281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":288219349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":422624,"kb_used_data":157488,"kb_used_omap":61,"kb_used_meta":264962,"kb_avail":660180256,"statfs":{"total":676457349120,"available":676024582144,"internally_reserved":0,"allocated":161267712,"data_stored":126347988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63095,"internal_metadata":271321481},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.870121"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1641,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-03-30T13:22:28.735303+0000","last_change":"2024-03-30T13:21:56.538529+0000","last_active":"2024-03-30T13:22:28.735303+0000","last_peered":"2024-03-30T13:22:28.735303+0000","last_clean":"2024-03-30T13:22:28.735303+0000","last_became_active":"2024-03-30T13:21:56.537232+0000","last_became_peered":"2024-03-30T13:21:56.537232+0000","last_unstale":"2024-03-30T13:22:28.735303+0000","last_undegraded":"2024-03-30T13:22:28.735303+0000","last_fullsized":"2024-03-30T13:22:28.735303+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:21:28.696384+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:21:28.696384+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:55.460477+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1587,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-03-30T13:22:28.386436+0000","last_change":"2024-03-30T13:22:28.004786+0000","last_active":"2024-03-30T13:22:28.386436+0000","last_peered":"2024-03-30T13:22:28.386436+0000","last_clean":"2024-03-30T13:22:28.386436+0000","last_became_active":"2024-03-30T13:22:15.770018+0000","last_became_peered":"2024-03-30T13:22:15.770018+0000","last_unstale":"2024-03-30T13:22:28.386436+0000","last_undegraded":"2024-03-30T13:22:28.386436+0000","last_fullsized":"2024-03-30T13:22:28.386436+0000","mapping_epoch":740,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":741,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:22:28.004525+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:22:28.004525+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:52.992325+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":339,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-03-30T13:22:28.860910+0000","last_change":"2024-03-30T13:22:07.121892+0000","last_active":"2024-03-30T13:22:28.860910+0000","last_peered":"2024-03-30T13:22:28.860910+0000","last_clean":"2024-03-30T13:22:28.860910+0000","last_became_active":"2024-03-30T13:20:14.422234+0000","last_became_peered":"2024-03-30T13:20:14.422234+0000","last_unstale":"2024-03-30T13:22:28.860910+0000","last_undegraded":"2024-03-30T13:22:28.860910+0000","last_fullsized":"2024-03-30T13:22:28.860910+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:22:07.121601+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:25.085038+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":352,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-03-30T13:22:28.860810+0000","last_change":"2024-03-30T13:22:27.910195+0000","last_active":"2024-03-30T13:22:28.860810+0000","last_peered":"2024-03-30T13:22:28.860810+0000","last_clean":"2024-03-30T13:22:28.860810+0000","last_became_active":"2024-03-30T13:21:56.541153+0000","last_became_peered":"2024-03-30T13:21:56.541153+0000","last_unstale":"2024-03-30T13:22:28.860810+0000","last_undegraded":"2024-03-30T13:22:28.860810+0000","last_fullsized":"2024-03-30T13:22:28.860810+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:22:27.909908+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:21:26.828701+0000","last_clean_scrub_stamp":"2024-03-30T13:22:27.909908+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:35.779509+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":523,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-03-30T13:22:28.861566+0000","last_change":"2024-03-30T13:21:56.308230+0000","last_active":"2024-03-30T13:22:28.861566+0000","last_peered":"2024-03-30T13:22:28.861566+0000","last_clean":"2024-03-30T13:22:28.861566+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:22:28.861566+0000","last_undegraded":"2024-03-30T13:22:28.861566+0000","last_fullsized":"2024-03-30T13:22:28.861566+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:56.307946+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:21:56.307946+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:22.988661+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":37,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-03-30T13:22:29.788285+0000","last_change":"2024-03-30T13:22:29.788365+0000","last_active":"2024-03-30T13:22:29.788285+0000","last_peered":"2024-03-30T13:22:29.788285+0000","last_clean":"2024-03-30T13:22:29.788285+0000","last_became_active":"2024-03-30T13:22:15.779048+0000","last_became_peered":"2024-03-30T13:22:15.779048+0000","last_unstale":"2024-03-30T13:22:29.788285+0000","last_undegraded":"2024-03-30T13:22:29.788285+0000","last_fullsized":"2024-03-30T13:22:29.788285+0000","mapping_epoch":740,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":741,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:22:29.788082+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:22:29.788082+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:48.271306+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11394981,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":387,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-03-30T13:22:28.860819+0000","last_change":"2024-03-30T13:21:57.102491+0000","last_active":"2024-03-30T13:22:28.860819+0000","last_peered":"2024-03-30T13:22:28.860819+0000","last_clean":"2024-03-30T13:22:28.860819+0000","last_became_active":"2024-03-30T13:20:14.421760+0000","last_became_peered":"2024-03-30T13:20:14.421760+0000","last_unstale":"2024-03-30T13:22:28.860819+0000","last_undegraded":"2024-03-30T13:22:28.860819+0000","last_fullsized":"2024-03-30T13:22:28.860819+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:21:57.102229+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:21:57.102229+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:21.918752+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":958,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-03-30T13:22:28.386379+0000","last_change":"2024-03-30T13:21:56.541906+0000","last_active":"2024-03-30T13:22:28.386379+0000","last_peered":"2024-03-30T13:22:28.386379+0000","last_clean":"2024-03-30T13:22:28.386379+0000","last_became_active":"2024-03-30T13:21:56.540619+0000","last_became_peered":"2024-03-30T13:21:56.540619+0000","last_unstale":"2024-03-30T13:22:28.386379+0000","last_undegraded":"2024-03-30T13:22:28.386379+0000","last_fullsized":"2024-03-30T13:22:28.386379+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:21:30.546139+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:21:30.546139+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:58.351721+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":73445600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":31,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-03-30T13:22:28.735721+0000","last_change":"2024-03-30T13:22:16.776395+0000","last_active":"2024-03-30T13:22:28.735721+0000","last_peered":"2024-03-30T13:22:28.735721+0000","last_clean":"2024-03-30T13:22:28.735721+0000","last_became_active":"2024-03-30T13:22:15.774709+0000","last_became_peered":"2024-03-30T13:22:15.774709+0000","last_unstale":"2024-03-30T13:22:28.735721+0000","last_undegraded":"2024-03-30T13:22:28.735721+0000","last_fullsized":"2024-03-30T13:22:28.735721+0000","mapping_epoch":740,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":741,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:22:16.776051+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:22:16.776051+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:35.676958+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":65983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1415,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-03-30T13:22:28.386390+0000","last_change":"2024-03-30T13:21:57.711005+0000","last_active":"2024-03-30T13:22:28.386390+0000","last_peered":"2024-03-30T13:22:28.386390+0000","last_clean":"2024-03-30T13:22:28.386390+0000","last_became_active":"2024-03-30T13:21:56.531330+0000","last_became_peered":"2024-03-30T13:21:56.531330+0000","last_unstale":"2024-03-30T13:22:28.386390+0000","last_undegraded":"2024-03-30T13:22:28.386390+0000","last_fullsized":"2024-03-30T13:22:28.386390+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:57.710868+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:21:57.710868+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:10.029079+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1668,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-03-30T13:22:28.735250+0000","last_change":"2024-03-30T13:21:57.935968+0000","last_active":"2024-03-30T13:22:28.735250+0000","last_peered":"2024-03-30T13:22:28.735250+0000","last_clean":"2024-03-30T13:22:28.735250+0000","last_became_active":"2024-03-30T13:21:56.537406+0000","last_became_peered":"2024-03-30T13:21:56.537406+0000","last_unstale":"2024-03-30T13:22:28.735250+0000","last_undegraded":"2024-03-30T13:22:28.735250+0000","last_fullsized":"2024-03-30T13:22:28.735250+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:21:57.935530+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:21:57.935530+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:14.327616+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1749,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-03-30T13:22:28.735679+0000","last_change":"2024-03-30T13:22:15.923801+0000","last_active":"2024-03-30T13:22:28.735679+0000","last_peered":"2024-03-30T13:22:28.735679+0000","last_clean":"2024-03-30T13:22:28.735679+0000","last_became_active":"2024-03-30T13:22:15.781029+0000","last_became_peered":"2024-03-30T13:22:15.781029+0000","last_unstale":"2024-03-30T13:22:28.735679+0000","last_undegraded":"2024-03-30T13:22:28.735679+0000","last_fullsized":"2024-03-30T13:22:28.735679+0000","mapping_epoch":740,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":741,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:21:49.068011+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:21:49.068011+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:09.174285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":572,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-03-30T13:22:28.861568+0000","last_change":"2024-03-30T13:22:16.261124+0000","last_active":"2024-03-30T13:22:28.861568+0000","last_peered":"2024-03-30T13:22:28.861568+0000","last_clean":"2024-03-30T13:22:28.861568+0000","last_became_active":"2024-03-30T13:22:15.780935+0000","last_became_peered":"2024-03-30T13:22:15.780935+0000","last_unstale":"2024-03-30T13:22:28.861568+0000","last_undegraded":"2024-03-30T13:22:28.861568+0000","last_fullsized":"2024-03-30T13:22:28.861568+0000","mapping_epoch":740,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":741,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:21:33.417986+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:21:33.417986+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:01.279294+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18319366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":342,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-03-30T13:22:28.860834+0000","last_change":"2024-03-30T13:22:15.796533+0000","last_active":"2024-03-30T13:22:28.860834+0000","last_peered":"2024-03-30T13:22:28.860834+0000","last_clean":"2024-03-30T13:22:28.860834+0000","last_became_active":"2024-03-30T13:22:15.774064+0000","last_became_peered":"2024-03-30T13:22:15.774064+0000","last_unstale":"2024-03-30T13:22:28.860834+0000","last_undegraded":"2024-03-30T13:22:28.860834+0000","last_fullsized":"2024-03-30T13:22:28.860834+0000","mapping_epoch":740,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":741,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:06.773612+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:21:06.773612+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:22:34.708160+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4974806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":563,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-03-30T13:22:28.861358+0000","last_change":"2024-03-30T13:22:18.351128+0000","last_active":"2024-03-30T13:22:28.861358+0000","last_peered":"2024-03-30T13:22:28.861358+0000","last_clean":"2024-03-30T13:22:28.861358+0000","last_became_active":"2024-03-30T13:21:56.534725+0000","last_became_peered":"2024-03-30T13:21:56.534725+0000","last_unstale":"2024-03-30T13:22:28.861358+0000","last_undegraded":"2024-03-30T13:22:28.861358+0000","last_fullsized":"2024-03-30T13:22:28.861358+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:22:18.350810+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:22:18.350810+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:29.264706+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":900,"reported_epoch":744,"state":"active+clean","last_fresh":"2024-03-30T13:22:28.386608+0000","last_change":"2024-03-30T13:21:59.583334+0000","last_active":"2024-03-30T13:22:28.386608+0000","last_peered":"2024-03-30T13:22:28.386608+0000","last_clean":"2024-03-30T13:22:28.386608+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:22:28.386608+0000","last_undegraded":"2024-03-30T13:22:28.386608+0000","last_fullsized":"2024-03-30T13:22:28.386608+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:21:59.583187+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:21:59.583187+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:19.245095+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":288219349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":110415872,"data_stored":110415872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387722,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66304,"kb_used_data":29104,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94305536,"statfs":{"total":96636764160,"available":96568868864,"internally_reserved":0,"allocated":29802496,"data_stored":24813964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157538,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66380,"kb_used_data":28732,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94305460,"statfs":{"total":96636764160,"available":96568791040,"internally_reserved":0,"allocated":29421568,"data_stored":24433036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":677,"seq":2907692859423,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62244,"kb_used_data":20436,"kb_used_omap":13,"kb_used_meta":41778,"kb_avail":94309596,"statfs":{"total":96636764160,"available":96573026304,"internally_reserved":0,"allocated":20926464,"data_stored":15937932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14287,"internal_metadata":42780721},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509769,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50548,"kb_used_data":15204,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94321292,"statfs":{"total":96636764160,"available":96585003008,"internally_reserved":0,"allocated":15568896,"data_stored":10580364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375999048,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52340,"kb_used_data":15140,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94319500,"statfs":{"total":96636764160,"available":96583168000,"internally_reserved":0,"allocated":15503360,"data_stored":10514828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509770,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65292,"kb_used_data":29948,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94306548,"statfs":{"total":96636764160,"available":96569905152,"internally_reserved":0,"allocated":30666752,"data_stored":25678220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742442,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59516,"kb_used_data":18924,"kb_used_omap":15,"kb_used_meta":40560,"kb_avail":94312324,"statfs":{"total":96636764160,"available":96575819776,"internally_reserved":0,"allocated":19378176,"data_stored":14389644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":41533891},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":21573632,"data_stored":21573632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6410240,"data_stored":6410240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20709376,"data_stored":20709376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:22:40.444 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:22:40.444 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:22:40.444 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-03-30T13:22:40.444 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-30T13:22:40.444 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-30T13:22:40.447 INFO:tasks.ceph.osd.3:Started 2024-03-30T13:22:40.447 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-30T13:22:40.499 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T13:22:40.495+0000 7fa730e607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:40.506 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T13:22:40.503+0000 7fa730e607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:40.506 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T13:22:40.503+0000 7fa730e607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:no valid command found; 10 closest matches: 2024-03-30T13:22:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:0 2024-03-30T13:22:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:1 2024-03-30T13:22:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:2 2024-03-30T13:22:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:abort 2024-03-30T13:22:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:assert 2024-03-30T13:22:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:config diff 2024-03-30T13:22:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:config diff get 2024-03-30T13:22:40.564 INFO:teuthology.orchestra.run.smithi067.stderr:config get 2024-03-30T13:22:40.564 INFO:teuthology.orchestra.run.smithi067.stderr:config help [] 2024-03-30T13:22:40.564 INFO:teuthology.orchestra.run.smithi067.stderr:config set ... 2024-03-30T13:22:40.564 INFO:teuthology.orchestra.run.smithi067.stderr:admin_socket: invalid command 2024-03-30T13:22:40.565 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:22:40.566 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-03-30T13:22:41.037 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T13:22:41.035+0000 7fa730e607c0 -1 Falling back to public interface 2024-03-30T13:22:44.786 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T13:22:44.783+0000 7fa730e607c0 -1 osd.3 738 log_to_monitors true 2024-03-30T13:22:45.508 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-03-30T13:22:45.507+0000 7fa728c35640 -1 osd.3 738 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:22:45.566 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-03-30T13:22:45.703 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-30T13:22:45.703 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-30T13:22:45.703 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-03-30T13:22:45.703 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-30T13:22:45.717 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:22:45.717 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-03-30T13:22:45.717 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 3 2024-03-30T13:22:45.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:45.835+0000 7f6d67275640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:45.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:45.851+0000 7f6d67275640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:46.034 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:22:46.031+0000 7fbb38f08640 -1 mon.a@0(leader).osd e746 definitely_dead 0 2024-03-30T13:22:46.470 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.3. 2024-03-30T13:22:46.503 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-03-30T13:22:46.504 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:22:46.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:46.619+0000 7fc609f95640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:46.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:46.635+0000 7fc609f95640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:47.472 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:22:47.504 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:22:47.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:47.623+0000 7fc2ab121640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:47.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:47.639+0000 7fc2ab121640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:48.481 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:22:48.510 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:22:48.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:48.631+0000 7f8042e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:48.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:48.647+0000 7f8042e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:49.508 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:22:49.533 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:22:49.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:49.651+0000 7f20f2788640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:49.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:49.667+0000 7f20f2788640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:50.525 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:22:50.548 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:22:50.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:50.663+0000 7fb294465640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:50.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:50.679+0000 7fb294465640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:51.530 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:22:51.554 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:22:51.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:51.671+0000 7f9f6f9f0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:51.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:51.687+0000 7f9f6f9f0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:52.553 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:22:52.576 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:22:52.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:52.695+0000 7f77e9ba5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:52.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:52.711+0000 7f77e9ba5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:53.549 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:22:53.577 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:22:53.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:53.695+0000 7f60514c3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:53.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:53.711+0000 7f60514c3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:54.572 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:22:54.599 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:22:54.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:54.715+0000 7f657e34c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:54.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:54.731+0000 7f657e34c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:55.586 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:22:55.609 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:22:55.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:55.731+0000 7ff1f2eb3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:55.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:55.747+0000 7ff1f2eb3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:56.595 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:22:56.622 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:22:56.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:56.735+0000 7fad653b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:56.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:56.751+0000 7fad653b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:57.614 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:22:57.638 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:22:57.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:57.759+0000 7f6cdbdb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:57.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:57.775+0000 7f6cdbdb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:58.618 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:22:58.644 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:22:58.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:58.763+0000 7fc4c7615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:58.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:58.779+0000 7fc4c7615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:59.629 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:22:59.653 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:22:59.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:59.771+0000 7f955d72d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:22:59.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:22:59.787+0000 7f955d72d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:00.641 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:23:00.665 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:23:00.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:00.779+0000 7fea799a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:00.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:00.795+0000 7fea799a3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:01.645 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:23:01.675 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:23:01.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:01.795+0000 7f2d4f688640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:02.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:02.011+0000 7f2d4cc23640 0 --2- 172.21.15.67:0/3400923238 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2d48143050 0x7f2d480a9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-03-30T13:23:02.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:02.015+0000 7f2d4f688640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:02.669 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:23:02.693 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:23:02.693 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:23:02.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:02.811+0000 7fd9f1c2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:02.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:02.823+0000 7fd9f1c2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:02.997 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:23:02.997 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:23:03.014 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1581,"stamp":"2024-03-30T13:23:02.667380+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":271748720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":462924,"kb_used_data":157788,"kb_used_omap":69,"kb_used_meta":301690,"kb_avail":754511796,"statfs":{"total":773094113280,"available":772620079104,"internally_reserved":0,"allocated":161574912,"data_stored":126152776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":70789,"internal_metadata":308931451},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.052778"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1679,"reported_epoch":760,"state":"active+clean","last_fresh":"2024-03-30T13:22:59.686438+0000","last_change":"2024-03-30T13:22:56.778810+0000","last_active":"2024-03-30T13:22:59.686438+0000","last_peered":"2024-03-30T13:22:59.686438+0000","last_clean":"2024-03-30T13:22:59.686438+0000","last_became_active":"2024-03-30T13:21:56.537232+0000","last_became_peered":"2024-03-30T13:21:56.537232+0000","last_unstale":"2024-03-30T13:22:59.686438+0000","last_undegraded":"2024-03-30T13:22:59.686438+0000","last_fullsized":"2024-03-30T13:22:59.686438+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:22:56.778530+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:22:56.778530+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:21.357262+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1621,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-03-30T13:22:57.617367+0000","last_change":"2024-03-30T13:22:47.493154+0000","last_active":"2024-03-30T13:22:57.617367+0000","last_peered":"2024-03-30T13:22:57.617367+0000","last_clean":"2024-03-30T13:22:57.617367+0000","last_became_active":"2024-03-30T13:22:47.491927+0000","last_became_peered":"2024-03-30T13:22:47.491927+0000","last_unstale":"2024-03-30T13:22:57.617367+0000","last_undegraded":"2024-03-30T13:22:57.617367+0000","last_fullsized":"2024-03-30T13:22:57.617367+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:22:28.004525+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:22:28.004525+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:30.107101+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":373,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-03-30T13:23:01.945832+0000","last_change":"2024-03-30T13:22:07.121892+0000","last_active":"2024-03-30T13:23:01.945832+0000","last_peered":"2024-03-30T13:23:01.945832+0000","last_clean":"2024-03-30T13:23:01.945832+0000","last_became_active":"2024-03-30T13:20:14.422234+0000","last_became_peered":"2024-03-30T13:20:14.422234+0000","last_unstale":"2024-03-30T13:23:01.945832+0000","last_undegraded":"2024-03-30T13:23:01.945832+0000","last_fullsized":"2024-03-30T13:23:01.945832+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:22:07.121601+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:25.085038+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":386,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-03-30T13:23:01.945767+0000","last_change":"2024-03-30T13:22:27.910195+0000","last_active":"2024-03-30T13:23:01.945767+0000","last_peered":"2024-03-30T13:23:01.945767+0000","last_clean":"2024-03-30T13:23:01.945767+0000","last_became_active":"2024-03-30T13:21:56.541153+0000","last_became_peered":"2024-03-30T13:21:56.541153+0000","last_unstale":"2024-03-30T13:23:01.945767+0000","last_undegraded":"2024-03-30T13:23:01.945767+0000","last_fullsized":"2024-03-30T13:23:01.945767+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:22:27.909908+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:21:26.828701+0000","last_clean_scrub_stamp":"2024-03-30T13:22:27.909908+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:35.779509+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":559,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-03-30T13:23:01.656394+0000","last_change":"2024-03-30T13:21:56.308230+0000","last_active":"2024-03-30T13:23:01.656394+0000","last_peered":"2024-03-30T13:23:01.656394+0000","last_clean":"2024-03-30T13:23:01.656394+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:23:01.656394+0000","last_undegraded":"2024-03-30T13:23:01.656394+0000","last_fullsized":"2024-03-30T13:23:01.656394+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:56.307946+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:21:56.307946+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:22.988661+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1627,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T13:23:00.646625+0000","last_change":"2024-03-30T13:22:47.492011+0000","last_active":"2024-03-30T13:23:00.646625+0000","last_peered":"2024-03-30T13:23:00.646625+0000","last_clean":"2024-03-30T13:23:00.646625+0000","last_became_active":"2024-03-30T13:22:47.491313+0000","last_became_peered":"2024-03-30T13:22:47.491313+0000","last_unstale":"2024-03-30T13:23:00.646625+0000","last_undegraded":"2024-03-30T13:23:00.646625+0000","last_fullsized":"2024-03-30T13:23:00.646625+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:22:29.788082+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:22:29.788082+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:39.706081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":421,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-03-30T13:23:01.945770+0000","last_change":"2024-03-30T13:21:57.102491+0000","last_active":"2024-03-30T13:23:01.945770+0000","last_peered":"2024-03-30T13:23:01.945770+0000","last_clean":"2024-03-30T13:23:01.945770+0000","last_became_active":"2024-03-30T13:20:14.421760+0000","last_became_peered":"2024-03-30T13:20:14.421760+0000","last_unstale":"2024-03-30T13:23:01.945770+0000","last_undegraded":"2024-03-30T13:23:01.945770+0000","last_fullsized":"2024-03-30T13:23:01.945770+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:21:57.102229+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:21:57.102229+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:21.918752+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":989,"reported_epoch":760,"state":"active+clean","last_fresh":"2024-03-30T13:22:59.633491+0000","last_change":"2024-03-30T13:21:56.541906+0000","last_active":"2024-03-30T13:22:59.633491+0000","last_peered":"2024-03-30T13:22:59.633491+0000","last_clean":"2024-03-30T13:22:59.633491+0000","last_became_active":"2024-03-30T13:21:56.540619+0000","last_became_peered":"2024-03-30T13:21:56.540619+0000","last_unstale":"2024-03-30T13:22:59.633491+0000","last_undegraded":"2024-03-30T13:22:59.633491+0000","last_fullsized":"2024-03-30T13:22:59.633491+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:21:30.546139+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:21:30.546139+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":73445600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1730,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T13:23:00.646426+0000","last_change":"2024-03-30T13:22:47.491942+0000","last_active":"2024-03-30T13:23:00.646426+0000","last_peered":"2024-03-30T13:23:00.646426+0000","last_clean":"2024-03-30T13:23:00.646426+0000","last_became_active":"2024-03-30T13:22:47.491356+0000","last_became_peered":"2024-03-30T13:22:47.491356+0000","last_unstale":"2024-03-30T13:23:00.646426+0000","last_undegraded":"2024-03-30T13:23:00.646426+0000","last_fullsized":"2024-03-30T13:23:00.646426+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:22:16.776051+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:22:16.776051+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:25.260656+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1443,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-03-30T13:22:57.617236+0000","last_change":"2024-03-30T13:21:57.711005+0000","last_active":"2024-03-30T13:22:57.617236+0000","last_peered":"2024-03-30T13:22:57.617236+0000","last_clean":"2024-03-30T13:22:57.617236+0000","last_became_active":"2024-03-30T13:21:56.531330+0000","last_became_peered":"2024-03-30T13:21:56.531330+0000","last_unstale":"2024-03-30T13:22:57.617236+0000","last_undegraded":"2024-03-30T13:22:57.617236+0000","last_fullsized":"2024-03-30T13:22:57.617236+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:57.710868+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:21:57.710868+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:10.029079+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1698,"reported_epoch":760,"state":"active+clean","last_fresh":"2024-03-30T13:22:59.686328+0000","last_change":"2024-03-30T13:21:57.935968+0000","last_active":"2024-03-30T13:22:59.686328+0000","last_peered":"2024-03-30T13:22:59.686328+0000","last_clean":"2024-03-30T13:22:59.686328+0000","last_became_active":"2024-03-30T13:21:56.537406+0000","last_became_peered":"2024-03-30T13:21:56.537406+0000","last_unstale":"2024-03-30T13:22:59.686328+0000","last_undegraded":"2024-03-30T13:22:59.686328+0000","last_fullsized":"2024-03-30T13:22:59.686328+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:21:57.935530+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:21:57.935530+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:14.327616+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1785,"reported_epoch":760,"state":"active+clean","last_fresh":"2024-03-30T13:22:59.686432+0000","last_change":"2024-03-30T13:22:47.489627+0000","last_active":"2024-03-30T13:22:59.686432+0000","last_peered":"2024-03-30T13:22:59.686432+0000","last_clean":"2024-03-30T13:22:59.686432+0000","last_became_active":"2024-03-30T13:22:47.488417+0000","last_became_peered":"2024-03-30T13:22:47.488417+0000","last_unstale":"2024-03-30T13:22:59.686432+0000","last_undegraded":"2024-03-30T13:22:59.686432+0000","last_fullsized":"2024-03-30T13:22:59.686432+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:21:49.068011+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:21:49.068011+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:15.253697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":621,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-03-30T13:23:01.656373+0000","last_change":"2024-03-30T13:22:53.439552+0000","last_active":"2024-03-30T13:23:01.656373+0000","last_peered":"2024-03-30T13:23:01.656373+0000","last_clean":"2024-03-30T13:23:01.656373+0000","last_became_active":"2024-03-30T13:22:47.493033+0000","last_became_peered":"2024-03-30T13:22:47.493033+0000","last_unstale":"2024-03-30T13:23:01.656373+0000","last_undegraded":"2024-03-30T13:23:01.656373+0000","last_fullsized":"2024-03-30T13:23:01.656373+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:22:53.439303+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:22:53.439303+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:10.009027+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18319366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":390,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-03-30T13:23:01.945798+0000","last_change":"2024-03-30T13:22:47.490299+0000","last_active":"2024-03-30T13:23:01.945798+0000","last_peered":"2024-03-30T13:23:01.945798+0000","last_clean":"2024-03-30T13:23:01.945798+0000","last_became_active":"2024-03-30T13:22:47.489008+0000","last_became_peered":"2024-03-30T13:22:47.489008+0000","last_unstale":"2024-03-30T13:23:01.945798+0000","last_undegraded":"2024-03-30T13:23:01.945798+0000","last_fullsized":"2024-03-30T13:23:01.945798+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:22:38.916929+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:22:38.916929+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:51.005597+0000","scrub_duration":883,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4974806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":598,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-03-30T13:23:01.656394+0000","last_change":"2024-03-30T13:22:18.351128+0000","last_active":"2024-03-30T13:23:01.656394+0000","last_peered":"2024-03-30T13:23:01.656394+0000","last_clean":"2024-03-30T13:23:01.656394+0000","last_became_active":"2024-03-30T13:21:56.534725+0000","last_became_peered":"2024-03-30T13:21:56.534725+0000","last_unstale":"2024-03-30T13:23:01.656394+0000","last_undegraded":"2024-03-30T13:23:01.656394+0000","last_fullsized":"2024-03-30T13:23:01.656394+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:22:18.350810+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:22:18.350810+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:29.264706+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":931,"reported_epoch":760,"state":"active+clean","last_fresh":"2024-03-30T13:22:59.633570+0000","last_change":"2024-03-30T13:21:59.583334+0000","last_active":"2024-03-30T13:22:59.633570+0000","last_peered":"2024-03-30T13:22:59.633570+0000","last_clean":"2024-03-30T13:22:59.633570+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:22:59.633570+0000","last_undegraded":"2024-03-30T13:22:59.633570+0000","last_fullsized":"2024-03-30T13:22:59.633570+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:21:59.583187+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:21:59.583187+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:19.245095+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":271748720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387727,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60716,"kb_used_data":23156,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94311124,"statfs":{"total":96636764160,"available":96574590976,"internally_reserved":0,"allocated":23711744,"data_stored":19208265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157543,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65744,"kb_used_data":27552,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94306096,"statfs":{"total":96636764160,"available":96569442304,"internally_reserved":0,"allocated":28213248,"data_stored":23904578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":677,"seq":2907692859427,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61564,"kb_used_data":19580,"kb_used_omap":13,"kb_used_meta":41778,"kb_avail":94310276,"statfs":{"total":96636764160,"available":96573722624,"internally_reserved":0,"allocated":20049920,"data_stored":15560043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14287,"internal_metadata":42780721},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509773,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50248,"kb_used_data":14360,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94321592,"statfs":{"total":96636764160,"available":96585310208,"internally_reserved":0,"allocated":14704640,"data_stored":10207962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375999052,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51380,"kb_used_data":13636,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94320460,"statfs":{"total":96636764160,"available":96584151040,"internally_reserved":0,"allocated":13963264,"data_stored":9466586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509774,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57072,"kb_used_data":21368,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94314768,"statfs":{"total":96636764160,"available":96578322432,"internally_reserved":0,"allocated":21880832,"data_stored":17478591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742447,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58708,"kb_used_data":17756,"kb_used_omap":15,"kb_used_meta":40560,"kb_avail":94313132,"statfs":{"total":96636764160,"available":96576647168,"internally_reserved":0,"allocated":18182144,"data_stored":13866673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":41533891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570117,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57492,"kb_used_data":20380,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94314348,"statfs":{"total":96636764160,"available":96577892352,"internally_reserved":0,"allocated":20869120,"data_stored":16460078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:23:03.015 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:23:03.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:03.131+0000 7f070722d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:03.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:03.147+0000 7f070722d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:03.315 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:23:03.316 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:23:03.332 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1581,"stamp":"2024-03-30T13:23:02.667380+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":271748720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":462924,"kb_used_data":157788,"kb_used_omap":69,"kb_used_meta":301690,"kb_avail":754511796,"statfs":{"total":773094113280,"available":772620079104,"internally_reserved":0,"allocated":161574912,"data_stored":126152776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":70789,"internal_metadata":308931451},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.052778"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1679,"reported_epoch":760,"state":"active+clean","last_fresh":"2024-03-30T13:22:59.686438+0000","last_change":"2024-03-30T13:22:56.778810+0000","last_active":"2024-03-30T13:22:59.686438+0000","last_peered":"2024-03-30T13:22:59.686438+0000","last_clean":"2024-03-30T13:22:59.686438+0000","last_became_active":"2024-03-30T13:21:56.537232+0000","last_became_peered":"2024-03-30T13:21:56.537232+0000","last_unstale":"2024-03-30T13:22:59.686438+0000","last_undegraded":"2024-03-30T13:22:59.686438+0000","last_fullsized":"2024-03-30T13:22:59.686438+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:22:56.778530+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:22:56.778530+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:21.357262+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1621,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-03-30T13:22:57.617367+0000","last_change":"2024-03-30T13:22:47.493154+0000","last_active":"2024-03-30T13:22:57.617367+0000","last_peered":"2024-03-30T13:22:57.617367+0000","last_clean":"2024-03-30T13:22:57.617367+0000","last_became_active":"2024-03-30T13:22:47.491927+0000","last_became_peered":"2024-03-30T13:22:47.491927+0000","last_unstale":"2024-03-30T13:22:57.617367+0000","last_undegraded":"2024-03-30T13:22:57.617367+0000","last_fullsized":"2024-03-30T13:22:57.617367+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:22:28.004525+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:22:28.004525+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:30.107101+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":373,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-03-30T13:23:01.945832+0000","last_change":"2024-03-30T13:22:07.121892+0000","last_active":"2024-03-30T13:23:01.945832+0000","last_peered":"2024-03-30T13:23:01.945832+0000","last_clean":"2024-03-30T13:23:01.945832+0000","last_became_active":"2024-03-30T13:20:14.422234+0000","last_became_peered":"2024-03-30T13:20:14.422234+0000","last_unstale":"2024-03-30T13:23:01.945832+0000","last_undegraded":"2024-03-30T13:23:01.945832+0000","last_fullsized":"2024-03-30T13:23:01.945832+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:22:07.121601+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:25.085038+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":386,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-03-30T13:23:01.945767+0000","last_change":"2024-03-30T13:22:27.910195+0000","last_active":"2024-03-30T13:23:01.945767+0000","last_peered":"2024-03-30T13:23:01.945767+0000","last_clean":"2024-03-30T13:23:01.945767+0000","last_became_active":"2024-03-30T13:21:56.541153+0000","last_became_peered":"2024-03-30T13:21:56.541153+0000","last_unstale":"2024-03-30T13:23:01.945767+0000","last_undegraded":"2024-03-30T13:23:01.945767+0000","last_fullsized":"2024-03-30T13:23:01.945767+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:22:27.909908+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:21:26.828701+0000","last_clean_scrub_stamp":"2024-03-30T13:22:27.909908+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:35.779509+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":559,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-03-30T13:23:01.656394+0000","last_change":"2024-03-30T13:21:56.308230+0000","last_active":"2024-03-30T13:23:01.656394+0000","last_peered":"2024-03-30T13:23:01.656394+0000","last_clean":"2024-03-30T13:23:01.656394+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:23:01.656394+0000","last_undegraded":"2024-03-30T13:23:01.656394+0000","last_fullsized":"2024-03-30T13:23:01.656394+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:56.307946+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:21:56.307946+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:22.988661+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1627,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T13:23:00.646625+0000","last_change":"2024-03-30T13:22:47.492011+0000","last_active":"2024-03-30T13:23:00.646625+0000","last_peered":"2024-03-30T13:23:00.646625+0000","last_clean":"2024-03-30T13:23:00.646625+0000","last_became_active":"2024-03-30T13:22:47.491313+0000","last_became_peered":"2024-03-30T13:22:47.491313+0000","last_unstale":"2024-03-30T13:23:00.646625+0000","last_undegraded":"2024-03-30T13:23:00.646625+0000","last_fullsized":"2024-03-30T13:23:00.646625+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:22:29.788082+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:22:29.788082+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:39.706081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":421,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-03-30T13:23:01.945770+0000","last_change":"2024-03-30T13:21:57.102491+0000","last_active":"2024-03-30T13:23:01.945770+0000","last_peered":"2024-03-30T13:23:01.945770+0000","last_clean":"2024-03-30T13:23:01.945770+0000","last_became_active":"2024-03-30T13:20:14.421760+0000","last_became_peered":"2024-03-30T13:20:14.421760+0000","last_unstale":"2024-03-30T13:23:01.945770+0000","last_undegraded":"2024-03-30T13:23:01.945770+0000","last_fullsized":"2024-03-30T13:23:01.945770+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:21:57.102229+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:21:57.102229+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:21.918752+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":989,"reported_epoch":760,"state":"active+clean","last_fresh":"2024-03-30T13:22:59.633491+0000","last_change":"2024-03-30T13:21:56.541906+0000","last_active":"2024-03-30T13:22:59.633491+0000","last_peered":"2024-03-30T13:22:59.633491+0000","last_clean":"2024-03-30T13:22:59.633491+0000","last_became_active":"2024-03-30T13:21:56.540619+0000","last_became_peered":"2024-03-30T13:21:56.540619+0000","last_unstale":"2024-03-30T13:22:59.633491+0000","last_undegraded":"2024-03-30T13:22:59.633491+0000","last_fullsized":"2024-03-30T13:22:59.633491+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:21:30.546139+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:21:30.546139+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":73445600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1730,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T13:23:00.646426+0000","last_change":"2024-03-30T13:22:47.491942+0000","last_active":"2024-03-30T13:23:00.646426+0000","last_peered":"2024-03-30T13:23:00.646426+0000","last_clean":"2024-03-30T13:23:00.646426+0000","last_became_active":"2024-03-30T13:22:47.491356+0000","last_became_peered":"2024-03-30T13:22:47.491356+0000","last_unstale":"2024-03-30T13:23:00.646426+0000","last_undegraded":"2024-03-30T13:23:00.646426+0000","last_fullsized":"2024-03-30T13:23:00.646426+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:22:16.776051+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:22:16.776051+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:25.260656+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1443,"reported_epoch":758,"state":"active+clean","last_fresh":"2024-03-30T13:22:57.617236+0000","last_change":"2024-03-30T13:21:57.711005+0000","last_active":"2024-03-30T13:22:57.617236+0000","last_peered":"2024-03-30T13:22:57.617236+0000","last_clean":"2024-03-30T13:22:57.617236+0000","last_became_active":"2024-03-30T13:21:56.531330+0000","last_became_peered":"2024-03-30T13:21:56.531330+0000","last_unstale":"2024-03-30T13:22:57.617236+0000","last_undegraded":"2024-03-30T13:22:57.617236+0000","last_fullsized":"2024-03-30T13:22:57.617236+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:57.710868+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:21:57.710868+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:10.029079+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1698,"reported_epoch":760,"state":"active+clean","last_fresh":"2024-03-30T13:22:59.686328+0000","last_change":"2024-03-30T13:21:57.935968+0000","last_active":"2024-03-30T13:22:59.686328+0000","last_peered":"2024-03-30T13:22:59.686328+0000","last_clean":"2024-03-30T13:22:59.686328+0000","last_became_active":"2024-03-30T13:21:56.537406+0000","last_became_peered":"2024-03-30T13:21:56.537406+0000","last_unstale":"2024-03-30T13:22:59.686328+0000","last_undegraded":"2024-03-30T13:22:59.686328+0000","last_fullsized":"2024-03-30T13:22:59.686328+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:21:57.935530+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:21:57.935530+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:14.327616+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1785,"reported_epoch":760,"state":"active+clean","last_fresh":"2024-03-30T13:22:59.686432+0000","last_change":"2024-03-30T13:22:47.489627+0000","last_active":"2024-03-30T13:22:59.686432+0000","last_peered":"2024-03-30T13:22:59.686432+0000","last_clean":"2024-03-30T13:22:59.686432+0000","last_became_active":"2024-03-30T13:22:47.488417+0000","last_became_peered":"2024-03-30T13:22:47.488417+0000","last_unstale":"2024-03-30T13:22:59.686432+0000","last_undegraded":"2024-03-30T13:22:59.686432+0000","last_fullsized":"2024-03-30T13:22:59.686432+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:21:49.068011+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:21:49.068011+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:15.253697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":621,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-03-30T13:23:01.656373+0000","last_change":"2024-03-30T13:22:53.439552+0000","last_active":"2024-03-30T13:23:01.656373+0000","last_peered":"2024-03-30T13:23:01.656373+0000","last_clean":"2024-03-30T13:23:01.656373+0000","last_became_active":"2024-03-30T13:22:47.493033+0000","last_became_peered":"2024-03-30T13:22:47.493033+0000","last_unstale":"2024-03-30T13:23:01.656373+0000","last_undegraded":"2024-03-30T13:23:01.656373+0000","last_fullsized":"2024-03-30T13:23:01.656373+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:22:53.439303+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:22:53.439303+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:10.009027+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18319366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":390,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-03-30T13:23:01.945798+0000","last_change":"2024-03-30T13:22:47.490299+0000","last_active":"2024-03-30T13:23:01.945798+0000","last_peered":"2024-03-30T13:23:01.945798+0000","last_clean":"2024-03-30T13:23:01.945798+0000","last_became_active":"2024-03-30T13:22:47.489008+0000","last_became_peered":"2024-03-30T13:22:47.489008+0000","last_unstale":"2024-03-30T13:23:01.945798+0000","last_undegraded":"2024-03-30T13:23:01.945798+0000","last_fullsized":"2024-03-30T13:23:01.945798+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:22:38.916929+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:22:38.916929+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:51.005597+0000","scrub_duration":883,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4974806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":598,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-03-30T13:23:01.656394+0000","last_change":"2024-03-30T13:22:18.351128+0000","last_active":"2024-03-30T13:23:01.656394+0000","last_peered":"2024-03-30T13:23:01.656394+0000","last_clean":"2024-03-30T13:23:01.656394+0000","last_became_active":"2024-03-30T13:21:56.534725+0000","last_became_peered":"2024-03-30T13:21:56.534725+0000","last_unstale":"2024-03-30T13:23:01.656394+0000","last_undegraded":"2024-03-30T13:23:01.656394+0000","last_fullsized":"2024-03-30T13:23:01.656394+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:22:18.350810+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:22:18.350810+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:29.264706+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":931,"reported_epoch":760,"state":"active+clean","last_fresh":"2024-03-30T13:22:59.633570+0000","last_change":"2024-03-30T13:21:59.583334+0000","last_active":"2024-03-30T13:22:59.633570+0000","last_peered":"2024-03-30T13:22:59.633570+0000","last_clean":"2024-03-30T13:22:59.633570+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:22:59.633570+0000","last_undegraded":"2024-03-30T13:22:59.633570+0000","last_fullsized":"2024-03-30T13:22:59.633570+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:21:59.583187+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:21:59.583187+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:19.245095+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":271748720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387727,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60716,"kb_used_data":23156,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94311124,"statfs":{"total":96636764160,"available":96574590976,"internally_reserved":0,"allocated":23711744,"data_stored":19208265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157543,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65744,"kb_used_data":27552,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94306096,"statfs":{"total":96636764160,"available":96569442304,"internally_reserved":0,"allocated":28213248,"data_stored":23904578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":677,"seq":2907692859427,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61564,"kb_used_data":19580,"kb_used_omap":13,"kb_used_meta":41778,"kb_avail":94310276,"statfs":{"total":96636764160,"available":96573722624,"internally_reserved":0,"allocated":20049920,"data_stored":15560043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14287,"internal_metadata":42780721},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509773,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50248,"kb_used_data":14360,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94321592,"statfs":{"total":96636764160,"available":96585310208,"internally_reserved":0,"allocated":14704640,"data_stored":10207962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375999052,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51380,"kb_used_data":13636,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94320460,"statfs":{"total":96636764160,"available":96584151040,"internally_reserved":0,"allocated":13963264,"data_stored":9466586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509774,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57072,"kb_used_data":21368,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94314768,"statfs":{"total":96636764160,"available":96578322432,"internally_reserved":0,"allocated":21880832,"data_stored":17478591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742447,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58708,"kb_used_data":17756,"kb_used_omap":15,"kb_used_meta":40560,"kb_avail":94313132,"statfs":{"total":96636764160,"available":96576647168,"internally_reserved":0,"allocated":18182144,"data_stored":13866673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":41533891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570117,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57492,"kb_used_data":20380,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94314348,"statfs":{"total":96636764160,"available":96577892352,"internally_reserved":0,"allocated":20869120,"data_stored":16460078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:23:03.333 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:23:03.333 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:23:03.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:03.435+0000 7f90a395f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:03.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:03.447+0000 7f90a395f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:03.617 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:23:03.617 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":763,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:23:02.658110+0000","last_up_change":"2024-03-30T13:22:46.469659+0000","last_in_change":"2024-03-30T13:22:46.469659+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":37,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":739,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":461,"last_clean_end":575,"up_from":581,"up_thru":722,"down_at":576,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6818","nonce":4030404750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6821","nonce":4030404750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6828","nonce":4030404750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":4030404750},{"type":"v1","addr":"172.21.15.67:6825","nonce":4030404750}]},"public_addr":"172.21.15.67:6818/4030404750","cluster_addr":"172.21.15.67:6821/4030404750","heartbeat_back_addr":"172.21.15.67:6828/4030404750","heartbeat_front_addr":"172.21.15.67:6825/4030404750","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":668,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":699,"last_clean_end":738,"up_from":747,"up_thru":747,"down_at":739,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6801","nonce":3161027348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6803","nonce":3161027348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6807","nonce":3161027348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6805","nonce":3161027348}]},"public_addr":"172.21.15.67:6801/3161027348","cluster_addr":"172.21.15.67:6803/3161027348","heartbeat_back_addr":"172.21.15.67:6807/3161027348","heartbeat_front_addr":"172.21.15.67:6805/3161027348","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":559,"last_clean_end":666,"up_from":677,"up_thru":747,"down_at":667,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1606011629},{"type":"v1","addr":"172.21.15.82:6803","nonce":1606011629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":1606011629},{"type":"v1","addr":"172.21.15.82:6807","nonce":1606011629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":1606011629},{"type":"v1","addr":"172.21.15.82:6815","nonce":1606011629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":1606011629},{"type":"v1","addr":"172.21.15.82:6811","nonce":1606011629}]},"public_addr":"172.21.15.82:6803/1606011629","cluster_addr":"172.21.15.82:6807/1606011629","heartbeat_back_addr":"172.21.15.82:6815/1606011629","heartbeat_front_addr":"172.21.15.82:6811/1606011629","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":488,"last_clean_end":506,"up_from":515,"up_thru":747,"down_at":507,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6817","nonce":292505453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6819","nonce":292505453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6823","nonce":292505453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6821","nonce":292505453}]},"public_addr":"172.21.15.82:6817/292505453","cluster_addr":"172.21.15.82:6819/292505453","heartbeat_back_addr":"172.21.15.82:6823/292505453","heartbeat_front_addr":"172.21.15.82:6821/292505453","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":629,"last_clean_end":715,"up_from":722,"up_thru":747,"down_at":716,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6801","nonce":326680821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6806","nonce":326680821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6814","nonce":326680821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6810","nonce":326680821}]},"public_addr":"172.21.15.82:6801/326680821","cluster_addr":"172.21.15.82:6806/326680821","heartbeat_back_addr":"172.21.15.82:6814/326680821","heartbeat_front_addr":"172.21.15.82:6810/326680821","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":604,"last_clean_end":645,"up_from":650,"up_thru":747,"down_at":646,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6825","nonce":1497758647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6827","nonce":1497758647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6831","nonce":1497758647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6829","nonce":1497758647}]},"public_addr":"172.21.15.82:6825/1497758647","cluster_addr":"172.21.15.82:6827/1497758647","heartbeat_back_addr":"172.21.15.82:6831/1497758647","heartbeat_front_addr":"172.21.15.82:6829/1497758647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:22:47.482020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:22:27.774020+0000","dead_epoch":576},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:22:17.819130+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:22:45.755217+0000","dead_epoch":739},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:21:44.551643+0000","dead_epoch":667},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:22:43.071176+0000","dead_epoch":507},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:22:26.942185+0000","dead_epoch":716},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:22:51.504399+0000","dead_epoch":646}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:23:03.634 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:23:03.635 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:23:03.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:03.747+0000 7f8c1079c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:03.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:03.763+0000 7f8c1079c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:03.938 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:23:03.957 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:23:04.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:04.063+0000 7f2b4075a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:04.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:04.079+0000 7f2b4075a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:04.258 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:23:04.259 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:23:04.277 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:23:04.277 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:23:04.277 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:23:04.278 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:23:04.278 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 5, 1, 7, 4, 6, 3] 2024-03-30T13:23:04.278 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:23:04.278 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:23:04.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:04.383+0000 7f4243e82640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:04.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:04.395+0000 7f4243e82640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:04.564 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:23:04.565 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:23:04.581 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1582,"stamp":"2024-03-30T13:23:04.260663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":271748720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":462452,"kb_used_data":156948,"kb_used_omap":69,"kb_used_meta":301690,"kb_avail":754512268,"statfs":{"total":773094113280,"available":772620562432,"internally_reserved":0,"allocated":160714752,"data_stored":125761880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":70789,"internal_metadata":308931451},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.634489"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1679,"reported_epoch":760,"state":"active+clean","last_fresh":"2024-03-30T13:22:59.686438+0000","last_change":"2024-03-30T13:22:56.778810+0000","last_active":"2024-03-30T13:22:59.686438+0000","last_peered":"2024-03-30T13:22:59.686438+0000","last_clean":"2024-03-30T13:22:59.686438+0000","last_became_active":"2024-03-30T13:21:56.537232+0000","last_became_peered":"2024-03-30T13:21:56.537232+0000","last_unstale":"2024-03-30T13:22:59.686438+0000","last_undegraded":"2024-03-30T13:22:59.686438+0000","last_fullsized":"2024-03-30T13:22:59.686438+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:22:56.778530+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:22:56.778530+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:21.357262+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1631,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-03-30T13:23:02.674494+0000","last_change":"2024-03-30T13:22:47.493154+0000","last_active":"2024-03-30T13:23:02.674494+0000","last_peered":"2024-03-30T13:23:02.674494+0000","last_clean":"2024-03-30T13:23:02.674494+0000","last_became_active":"2024-03-30T13:22:47.491927+0000","last_became_peered":"2024-03-30T13:22:47.491927+0000","last_unstale":"2024-03-30T13:23:02.674494+0000","last_undegraded":"2024-03-30T13:23:02.674494+0000","last_fullsized":"2024-03-30T13:23:02.674494+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:22:28.004525+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:22:28.004525+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:30.107101+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":373,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-03-30T13:23:01.945832+0000","last_change":"2024-03-30T13:22:07.121892+0000","last_active":"2024-03-30T13:23:01.945832+0000","last_peered":"2024-03-30T13:23:01.945832+0000","last_clean":"2024-03-30T13:23:01.945832+0000","last_became_active":"2024-03-30T13:20:14.422234+0000","last_became_peered":"2024-03-30T13:20:14.422234+0000","last_unstale":"2024-03-30T13:23:01.945832+0000","last_undegraded":"2024-03-30T13:23:01.945832+0000","last_fullsized":"2024-03-30T13:23:01.945832+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:22:07.121601+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:25.085038+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":386,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-03-30T13:23:01.945767+0000","last_change":"2024-03-30T13:22:27.910195+0000","last_active":"2024-03-30T13:23:01.945767+0000","last_peered":"2024-03-30T13:23:01.945767+0000","last_clean":"2024-03-30T13:23:01.945767+0000","last_became_active":"2024-03-30T13:21:56.541153+0000","last_became_peered":"2024-03-30T13:21:56.541153+0000","last_unstale":"2024-03-30T13:23:01.945767+0000","last_undegraded":"2024-03-30T13:23:01.945767+0000","last_fullsized":"2024-03-30T13:23:01.945767+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:22:27.909908+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:21:26.828701+0000","last_clean_scrub_stamp":"2024-03-30T13:22:27.909908+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:35.779509+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":559,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-03-30T13:23:01.656394+0000","last_change":"2024-03-30T13:21:56.308230+0000","last_active":"2024-03-30T13:23:01.656394+0000","last_peered":"2024-03-30T13:23:01.656394+0000","last_clean":"2024-03-30T13:23:01.656394+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:23:01.656394+0000","last_undegraded":"2024-03-30T13:23:01.656394+0000","last_fullsized":"2024-03-30T13:23:01.656394+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:56.307946+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:21:56.307946+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:22.988661+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1627,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T13:23:00.646625+0000","last_change":"2024-03-30T13:22:47.492011+0000","last_active":"2024-03-30T13:23:00.646625+0000","last_peered":"2024-03-30T13:23:00.646625+0000","last_clean":"2024-03-30T13:23:00.646625+0000","last_became_active":"2024-03-30T13:22:47.491313+0000","last_became_peered":"2024-03-30T13:22:47.491313+0000","last_unstale":"2024-03-30T13:23:00.646625+0000","last_undegraded":"2024-03-30T13:23:00.646625+0000","last_fullsized":"2024-03-30T13:23:00.646625+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:22:29.788082+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:22:29.788082+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:39.706081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":421,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-03-30T13:23:01.945770+0000","last_change":"2024-03-30T13:21:57.102491+0000","last_active":"2024-03-30T13:23:01.945770+0000","last_peered":"2024-03-30T13:23:01.945770+0000","last_clean":"2024-03-30T13:23:01.945770+0000","last_became_active":"2024-03-30T13:20:14.421760+0000","last_became_peered":"2024-03-30T13:20:14.421760+0000","last_unstale":"2024-03-30T13:23:01.945770+0000","last_undegraded":"2024-03-30T13:23:01.945770+0000","last_fullsized":"2024-03-30T13:23:01.945770+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:21:57.102229+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:21:57.102229+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:21.918752+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":989,"reported_epoch":760,"state":"active+clean","last_fresh":"2024-03-30T13:22:59.633491+0000","last_change":"2024-03-30T13:21:56.541906+0000","last_active":"2024-03-30T13:22:59.633491+0000","last_peered":"2024-03-30T13:22:59.633491+0000","last_clean":"2024-03-30T13:22:59.633491+0000","last_became_active":"2024-03-30T13:21:56.540619+0000","last_became_peered":"2024-03-30T13:21:56.540619+0000","last_unstale":"2024-03-30T13:22:59.633491+0000","last_undegraded":"2024-03-30T13:22:59.633491+0000","last_fullsized":"2024-03-30T13:22:59.633491+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:21:30.546139+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:21:30.546139+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":73445600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1730,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T13:23:00.646426+0000","last_change":"2024-03-30T13:22:47.491942+0000","last_active":"2024-03-30T13:23:00.646426+0000","last_peered":"2024-03-30T13:23:00.646426+0000","last_clean":"2024-03-30T13:23:00.646426+0000","last_became_active":"2024-03-30T13:22:47.491356+0000","last_became_peered":"2024-03-30T13:22:47.491356+0000","last_unstale":"2024-03-30T13:23:00.646426+0000","last_undegraded":"2024-03-30T13:23:00.646426+0000","last_fullsized":"2024-03-30T13:23:00.646426+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:22:16.776051+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:22:16.776051+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:25.260656+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1453,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-03-30T13:23:02.674345+0000","last_change":"2024-03-30T13:21:57.711005+0000","last_active":"2024-03-30T13:23:02.674345+0000","last_peered":"2024-03-30T13:23:02.674345+0000","last_clean":"2024-03-30T13:23:02.674345+0000","last_became_active":"2024-03-30T13:21:56.531330+0000","last_became_peered":"2024-03-30T13:21:56.531330+0000","last_unstale":"2024-03-30T13:23:02.674345+0000","last_undegraded":"2024-03-30T13:23:02.674345+0000","last_fullsized":"2024-03-30T13:23:02.674345+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:57.710868+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:21:57.710868+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:10.029079+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1698,"reported_epoch":760,"state":"active+clean","last_fresh":"2024-03-30T13:22:59.686328+0000","last_change":"2024-03-30T13:21:57.935968+0000","last_active":"2024-03-30T13:22:59.686328+0000","last_peered":"2024-03-30T13:22:59.686328+0000","last_clean":"2024-03-30T13:22:59.686328+0000","last_became_active":"2024-03-30T13:21:56.537406+0000","last_became_peered":"2024-03-30T13:21:56.537406+0000","last_unstale":"2024-03-30T13:22:59.686328+0000","last_undegraded":"2024-03-30T13:22:59.686328+0000","last_fullsized":"2024-03-30T13:22:59.686328+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:21:57.935530+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:21:57.935530+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:14.327616+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1785,"reported_epoch":760,"state":"active+clean","last_fresh":"2024-03-30T13:22:59.686432+0000","last_change":"2024-03-30T13:22:47.489627+0000","last_active":"2024-03-30T13:22:59.686432+0000","last_peered":"2024-03-30T13:22:59.686432+0000","last_clean":"2024-03-30T13:22:59.686432+0000","last_became_active":"2024-03-30T13:22:47.488417+0000","last_became_peered":"2024-03-30T13:22:47.488417+0000","last_unstale":"2024-03-30T13:22:59.686432+0000","last_undegraded":"2024-03-30T13:22:59.686432+0000","last_fullsized":"2024-03-30T13:22:59.686432+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:21:49.068011+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:21:49.068011+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:15.253697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":621,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-03-30T13:23:01.656373+0000","last_change":"2024-03-30T13:22:53.439552+0000","last_active":"2024-03-30T13:23:01.656373+0000","last_peered":"2024-03-30T13:23:01.656373+0000","last_clean":"2024-03-30T13:23:01.656373+0000","last_became_active":"2024-03-30T13:22:47.493033+0000","last_became_peered":"2024-03-30T13:22:47.493033+0000","last_unstale":"2024-03-30T13:23:01.656373+0000","last_undegraded":"2024-03-30T13:23:01.656373+0000","last_fullsized":"2024-03-30T13:23:01.656373+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:22:53.439303+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:22:53.439303+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:10.009027+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18319366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":390,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-03-30T13:23:01.945798+0000","last_change":"2024-03-30T13:22:47.490299+0000","last_active":"2024-03-30T13:23:01.945798+0000","last_peered":"2024-03-30T13:23:01.945798+0000","last_clean":"2024-03-30T13:23:01.945798+0000","last_became_active":"2024-03-30T13:22:47.489008+0000","last_became_peered":"2024-03-30T13:22:47.489008+0000","last_unstale":"2024-03-30T13:23:01.945798+0000","last_undegraded":"2024-03-30T13:23:01.945798+0000","last_fullsized":"2024-03-30T13:23:01.945798+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:22:38.916929+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:22:38.916929+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:51.005597+0000","scrub_duration":883,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4974806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":598,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-03-30T13:23:01.656394+0000","last_change":"2024-03-30T13:22:18.351128+0000","last_active":"2024-03-30T13:23:01.656394+0000","last_peered":"2024-03-30T13:23:01.656394+0000","last_clean":"2024-03-30T13:23:01.656394+0000","last_became_active":"2024-03-30T13:21:56.534725+0000","last_became_peered":"2024-03-30T13:21:56.534725+0000","last_unstale":"2024-03-30T13:23:01.656394+0000","last_undegraded":"2024-03-30T13:23:01.656394+0000","last_fullsized":"2024-03-30T13:23:01.656394+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:22:18.350810+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:22:18.350810+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:29.264706+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":931,"reported_epoch":760,"state":"active+clean","last_fresh":"2024-03-30T13:22:59.633570+0000","last_change":"2024-03-30T13:21:59.583334+0000","last_active":"2024-03-30T13:22:59.633570+0000","last_peered":"2024-03-30T13:22:59.633570+0000","last_clean":"2024-03-30T13:22:59.633570+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:22:59.633570+0000","last_undegraded":"2024-03-30T13:22:59.633570+0000","last_fullsized":"2024-03-30T13:22:59.633570+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:21:59.583187+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:21:59.583187+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:19.245095+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":271748720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387727,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60716,"kb_used_data":23156,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94311124,"statfs":{"total":96636764160,"available":96574590976,"internally_reserved":0,"allocated":23711744,"data_stored":19208265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157543,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65744,"kb_used_data":27552,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94306096,"statfs":{"total":96636764160,"available":96569442304,"internally_reserved":0,"allocated":28213248,"data_stored":23904578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":677,"seq":2907692859428,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61092,"kb_used_data":18740,"kb_used_omap":13,"kb_used_meta":41778,"kb_avail":94310748,"statfs":{"total":96636764160,"available":96574205952,"internally_reserved":0,"allocated":19189760,"data_stored":15169147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14287,"internal_metadata":42780721},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509773,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50248,"kb_used_data":14360,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94321592,"statfs":{"total":96636764160,"available":96585310208,"internally_reserved":0,"allocated":14704640,"data_stored":10207962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":581,"seq":2495375999052,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51380,"kb_used_data":13636,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94320460,"statfs":{"total":96636764160,"available":96584151040,"internally_reserved":0,"allocated":13963264,"data_stored":9466586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9339,"internal_metadata":38067077},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509774,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57072,"kb_used_data":21368,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94314768,"statfs":{"total":96636764160,"available":96578322432,"internally_reserved":0,"allocated":21880832,"data_stored":17478591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742447,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58708,"kb_used_data":17756,"kb_used_omap":15,"kb_used_meta":40560,"kb_avail":94313132,"statfs":{"total":96636764160,"available":96576647168,"internally_reserved":0,"allocated":18182144,"data_stored":13866673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":41533891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570117,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57492,"kb_used_data":20380,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94314348,"statfs":{"total":96636764160,"available":96577892352,"internally_reserved":0,"allocated":20869120,"data_stored":16460078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:23:04.582 INFO:tasks.thrashosds.thrasher:Choosing PG 1.3 with acting set [1, 6, 5] 2024-03-30T13:23:04.582 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 2, 5, 1, 7, 4, 6, 3] 2024-03-30T13:23:04.583 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-30T13:23:04.583 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:23:04.681 INFO:tasks.ceph.osd.1:Stopped 2024-03-30T13:23:04.681 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 2, 5, 1, 7, 4, 6, 3] 2024-03-30T13:23:04.682 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 1 2024-03-30T13:23:04.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:04.803+0000 7fb3a0177640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:04.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:04.819+0000 7fb3a0177640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:04.996 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:23:04.995+0000 7fbb38f08640 -1 mon.a@0(leader).osd e763 definitely_dead 0 2024-03-30T13:23:05.668 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.1. 2024-03-30T13:23:05.696 INFO:tasks.thrashosds.thrasher:dead_osds=[1], live_osds=[0, 2, 5, 7, 4, 6, 3] 2024-03-30T13:23:05.696 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:23:05.696 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:23:05.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:05.811+0000 7fa9b2059640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:05.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:05.823+0000 7fa9b2059640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:06.000 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:23:06.000 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:23:06.017 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1583,"stamp":"2024-03-30T13:23:05.672225+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":271748720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":45,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":409024,"kb_used_data":141632,"kb_used_omap":60,"kb_used_meta":264515,"kb_avail":660193856,"statfs":{"total":676457349120,"available":676038508544,"internally_reserved":0,"allocated":145031168,"data_stored":115514398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61450,"internal_metadata":270864374},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.412550"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1685,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-03-30T13:23:03.048393+0000","last_change":"2024-03-30T13:22:56.778810+0000","last_active":"2024-03-30T13:23:03.048393+0000","last_peered":"2024-03-30T13:23:03.048393+0000","last_clean":"2024-03-30T13:23:03.048393+0000","last_became_active":"2024-03-30T13:21:56.537232+0000","last_became_peered":"2024-03-30T13:21:56.537232+0000","last_unstale":"2024-03-30T13:23:03.048393+0000","last_undegraded":"2024-03-30T13:23:03.048393+0000","last_fullsized":"2024-03-30T13:23:03.048393+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:22:56.778530+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:22:56.778530+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:21.357262+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1631,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-03-30T13:23:02.674494+0000","last_change":"2024-03-30T13:22:47.493154+0000","last_active":"2024-03-30T13:23:02.674494+0000","last_peered":"2024-03-30T13:23:02.674494+0000","last_clean":"2024-03-30T13:23:02.674494+0000","last_became_active":"2024-03-30T13:22:47.491927+0000","last_became_peered":"2024-03-30T13:22:47.491927+0000","last_unstale":"2024-03-30T13:23:02.674494+0000","last_undegraded":"2024-03-30T13:23:02.674494+0000","last_fullsized":"2024-03-30T13:23:02.674494+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:22:28.004525+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:22:28.004525+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:30.107101+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":373,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-03-30T13:23:01.945832+0000","last_change":"2024-03-30T13:22:07.121892+0000","last_active":"2024-03-30T13:23:01.945832+0000","last_peered":"2024-03-30T13:23:01.945832+0000","last_clean":"2024-03-30T13:23:01.945832+0000","last_became_active":"2024-03-30T13:20:14.422234+0000","last_became_peered":"2024-03-30T13:20:14.422234+0000","last_unstale":"2024-03-30T13:23:01.945832+0000","last_undegraded":"2024-03-30T13:23:01.945832+0000","last_fullsized":"2024-03-30T13:23:01.945832+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:22:07.121601+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:25.085038+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":386,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-03-30T13:23:01.945767+0000","last_change":"2024-03-30T13:22:27.910195+0000","last_active":"2024-03-30T13:23:01.945767+0000","last_peered":"2024-03-30T13:23:01.945767+0000","last_clean":"2024-03-30T13:23:01.945767+0000","last_became_active":"2024-03-30T13:21:56.541153+0000","last_became_peered":"2024-03-30T13:21:56.541153+0000","last_unstale":"2024-03-30T13:23:01.945767+0000","last_undegraded":"2024-03-30T13:23:01.945767+0000","last_fullsized":"2024-03-30T13:23:01.945767+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:22:27.909908+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:21:26.828701+0000","last_clean_scrub_stamp":"2024-03-30T13:22:27.909908+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:35.779509+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":559,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-03-30T13:23:01.656394+0000","last_change":"2024-03-30T13:21:56.308230+0000","last_active":"2024-03-30T13:23:01.656394+0000","last_peered":"2024-03-30T13:23:01.656394+0000","last_clean":"2024-03-30T13:23:01.656394+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:23:01.656394+0000","last_undegraded":"2024-03-30T13:23:01.656394+0000","last_fullsized":"2024-03-30T13:23:01.656394+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:56.307946+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:21:56.307946+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:22.988661+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1627,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T13:23:00.646625+0000","last_change":"2024-03-30T13:22:47.492011+0000","last_active":"2024-03-30T13:23:00.646625+0000","last_peered":"2024-03-30T13:23:00.646625+0000","last_clean":"2024-03-30T13:23:00.646625+0000","last_became_active":"2024-03-30T13:22:47.491313+0000","last_became_peered":"2024-03-30T13:22:47.491313+0000","last_unstale":"2024-03-30T13:23:00.646625+0000","last_undegraded":"2024-03-30T13:23:00.646625+0000","last_fullsized":"2024-03-30T13:23:00.646625+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:22:29.788082+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:22:29.788082+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:39.706081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":421,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-03-30T13:23:01.945770+0000","last_change":"2024-03-30T13:21:57.102491+0000","last_active":"2024-03-30T13:23:01.945770+0000","last_peered":"2024-03-30T13:23:01.945770+0000","last_clean":"2024-03-30T13:23:01.945770+0000","last_became_active":"2024-03-30T13:20:14.421760+0000","last_became_peered":"2024-03-30T13:20:14.421760+0000","last_unstale":"2024-03-30T13:23:01.945770+0000","last_undegraded":"2024-03-30T13:23:01.945770+0000","last_fullsized":"2024-03-30T13:23:01.945770+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:21:57.102229+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:21:57.102229+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:21.918752+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":989,"reported_epoch":760,"state":"stale+active+clean","last_fresh":"2024-03-30T13:22:59.633491+0000","last_change":"2024-03-30T13:21:56.541906+0000","last_active":"2024-03-30T13:22:59.633491+0000","last_peered":"2024-03-30T13:22:59.633491+0000","last_clean":"2024-03-30T13:22:59.633491+0000","last_became_active":"2024-03-30T13:21:56.540619+0000","last_became_peered":"2024-03-30T13:21:56.540619+0000","last_unstale":"2024-03-30T13:23:05.672249+0000","last_undegraded":"2024-03-30T13:22:59.633491+0000","last_fullsized":"2024-03-30T13:22:59.633491+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:21:30.546139+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:21:30.546139+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":73445600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1730,"reported_epoch":761,"state":"active+clean","last_fresh":"2024-03-30T13:23:00.646426+0000","last_change":"2024-03-30T13:22:47.491942+0000","last_active":"2024-03-30T13:23:00.646426+0000","last_peered":"2024-03-30T13:23:00.646426+0000","last_clean":"2024-03-30T13:23:00.646426+0000","last_became_active":"2024-03-30T13:22:47.491356+0000","last_became_peered":"2024-03-30T13:22:47.491356+0000","last_unstale":"2024-03-30T13:23:00.646426+0000","last_undegraded":"2024-03-30T13:23:00.646426+0000","last_fullsized":"2024-03-30T13:23:00.646426+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:22:16.776051+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:22:16.776051+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:25.260656+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1453,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-03-30T13:23:02.674345+0000","last_change":"2024-03-30T13:21:57.711005+0000","last_active":"2024-03-30T13:23:02.674345+0000","last_peered":"2024-03-30T13:23:02.674345+0000","last_clean":"2024-03-30T13:23:02.674345+0000","last_became_active":"2024-03-30T13:21:56.531330+0000","last_became_peered":"2024-03-30T13:21:56.531330+0000","last_unstale":"2024-03-30T13:23:02.674345+0000","last_undegraded":"2024-03-30T13:23:02.674345+0000","last_fullsized":"2024-03-30T13:23:02.674345+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:21:57.710868+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:21:57.710868+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:10.029079+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1704,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-03-30T13:23:03.048349+0000","last_change":"2024-03-30T13:21:57.935968+0000","last_active":"2024-03-30T13:23:03.048349+0000","last_peered":"2024-03-30T13:23:03.048349+0000","last_clean":"2024-03-30T13:23:03.048349+0000","last_became_active":"2024-03-30T13:21:56.537406+0000","last_became_peered":"2024-03-30T13:21:56.537406+0000","last_unstale":"2024-03-30T13:23:03.048349+0000","last_undegraded":"2024-03-30T13:23:03.048349+0000","last_fullsized":"2024-03-30T13:23:03.048349+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:21:57.935530+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:21:57.935530+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:14.327616+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1791,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-03-30T13:23:03.048503+0000","last_change":"2024-03-30T13:22:47.489627+0000","last_active":"2024-03-30T13:23:03.048503+0000","last_peered":"2024-03-30T13:23:03.048503+0000","last_clean":"2024-03-30T13:23:03.048503+0000","last_became_active":"2024-03-30T13:22:47.488417+0000","last_became_peered":"2024-03-30T13:22:47.488417+0000","last_unstale":"2024-03-30T13:23:03.048503+0000","last_undegraded":"2024-03-30T13:23:03.048503+0000","last_fullsized":"2024-03-30T13:23:03.048503+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:21:49.068011+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:15:12.092921+0000","last_clean_scrub_stamp":"2024-03-30T13:21:49.068011+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:15.253697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":621,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-03-30T13:23:01.656373+0000","last_change":"2024-03-30T13:22:53.439552+0000","last_active":"2024-03-30T13:23:01.656373+0000","last_peered":"2024-03-30T13:23:01.656373+0000","last_clean":"2024-03-30T13:23:01.656373+0000","last_became_active":"2024-03-30T13:22:47.493033+0000","last_became_peered":"2024-03-30T13:22:47.493033+0000","last_unstale":"2024-03-30T13:23:01.656373+0000","last_undegraded":"2024-03-30T13:23:01.656373+0000","last_fullsized":"2024-03-30T13:23:01.656373+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:22:53.439303+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:22:53.439303+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:10.009027+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18319366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":390,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-03-30T13:23:01.945798+0000","last_change":"2024-03-30T13:22:47.490299+0000","last_active":"2024-03-30T13:23:01.945798+0000","last_peered":"2024-03-30T13:23:01.945798+0000","last_clean":"2024-03-30T13:23:01.945798+0000","last_became_active":"2024-03-30T13:22:47.489008+0000","last_became_peered":"2024-03-30T13:22:47.489008+0000","last_unstale":"2024-03-30T13:23:01.945798+0000","last_undegraded":"2024-03-30T13:23:01.945798+0000","last_fullsized":"2024-03-30T13:23:01.945798+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:22:38.916929+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:22:38.916929+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:51.005597+0000","scrub_duration":883,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4974806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":598,"reported_epoch":762,"state":"active+clean","last_fresh":"2024-03-30T13:23:01.656394+0000","last_change":"2024-03-30T13:22:18.351128+0000","last_active":"2024-03-30T13:23:01.656394+0000","last_peered":"2024-03-30T13:23:01.656394+0000","last_clean":"2024-03-30T13:23:01.656394+0000","last_became_active":"2024-03-30T13:21:56.534725+0000","last_became_peered":"2024-03-30T13:21:56.534725+0000","last_unstale":"2024-03-30T13:23:01.656394+0000","last_undegraded":"2024-03-30T13:23:01.656394+0000","last_fullsized":"2024-03-30T13:23:01.656394+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:22:18.350810+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:22:18.350810+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:29.264706+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":931,"reported_epoch":760,"state":"stale+active+clean","last_fresh":"2024-03-30T13:22:59.633570+0000","last_change":"2024-03-30T13:21:59.583334+0000","last_active":"2024-03-30T13:22:59.633570+0000","last_peered":"2024-03-30T13:22:59.633570+0000","last_clean":"2024-03-30T13:22:59.633570+0000","last_became_active":"2024-03-30T13:16:50.991349+0000","last_became_peered":"2024-03-30T13:16:50.991349+0000","last_unstale":"2024-03-30T13:23:05.672253+0000","last_undegraded":"2024-03-30T13:22:59.633570+0000","last_fullsized":"2024-03-30T13:22:59.633570+0000","mapping_epoch":581,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":582,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:21:59.583187+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:21:59.583187+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:19.245095+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":271748720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387728,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60396,"kb_used_data":22652,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94311444,"statfs":{"total":96636764160,"available":96574918656,"internally_reserved":0,"allocated":23195648,"data_stored":18973396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157543,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65744,"kb_used_data":27552,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94306096,"statfs":{"total":96636764160,"available":96569442304,"internally_reserved":0,"allocated":28213248,"data_stored":23904578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":677,"seq":2907692859428,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61092,"kb_used_data":18740,"kb_used_omap":13,"kb_used_meta":41778,"kb_avail":94310748,"statfs":{"total":96636764160,"available":96574205952,"internally_reserved":0,"allocated":19189760,"data_stored":15169147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14287,"internal_metadata":42780721},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509774,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49208,"kb_used_data":13688,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94322632,"statfs":{"total":96636764160,"available":96586375168,"internally_reserved":0,"allocated":14016512,"data_stored":9896151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509775,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56384,"kb_used_data":20864,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315456,"statfs":{"total":96636764160,"available":96579026944,"internally_reserved":0,"allocated":21364736,"data_stored":17244375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742447,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58708,"kb_used_data":17756,"kb_used_omap":15,"kb_used_meta":40560,"kb_avail":94313132,"statfs":{"total":96636764160,"available":96576647168,"internally_reserved":0,"allocated":18182144,"data_stored":13866673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":41533891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570117,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57492,"kb_used_data":20380,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94314348,"statfs":{"total":96636764160,"available":96577892352,"internally_reserved":0,"allocated":20869120,"data_stored":16460078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:23:06.019 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:23:31.019 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:23:31.020 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:23:31.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:31.146+0000 7f86da597640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:31.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:31.162+0000 7f86da597640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:31.333 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:23:31.334 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:23:31.351 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1598,"stamp":"2024-03-30T13:23:30.267954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":270101667,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":408044,"kb_used_data":143044,"kb_used_omap":60,"kb_used_meta":264515,"kb_avail":660194836,"statfs":{"total":676457349120,"available":676039512064,"internally_reserved":0,"allocated":146477056,"data_stored":119299875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":61450,"internal_metadata":270864374},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003480"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1691,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-03-30T13:23:08.573749+0000","last_change":"2024-03-30T13:22:56.778810+0000","last_active":"2024-03-30T13:23:08.573749+0000","last_peered":"2024-03-30T13:23:08.573749+0000","last_clean":"2024-03-30T13:23:08.573749+0000","last_became_active":"2024-03-30T13:21:56.537232+0000","last_became_peered":"2024-03-30T13:21:56.537232+0000","last_unstale":"2024-03-30T13:23:08.573749+0000","last_undegraded":"2024-03-30T13:23:08.573749+0000","last_fullsized":"2024-03-30T13:23:08.573749+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:22:56.778530+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:22:56.778530+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:21.357262+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1637,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-03-30T13:23:07.709086+0000","last_change":"2024-03-30T13:22:47.493154+0000","last_active":"2024-03-30T13:23:07.709086+0000","last_peered":"2024-03-30T13:23:07.709086+0000","last_clean":"2024-03-30T13:23:07.709086+0000","last_became_active":"2024-03-30T13:22:47.491927+0000","last_became_peered":"2024-03-30T13:22:47.491927+0000","last_unstale":"2024-03-30T13:23:07.709086+0000","last_undegraded":"2024-03-30T13:23:07.709086+0000","last_fullsized":"2024-03-30T13:23:07.709086+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:22:28.004525+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:22:28.004525+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:30.107101+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":403,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-03-30T13:23:20.324343+0000","last_change":"2024-03-30T13:23:20.324417+0000","last_active":"2024-03-30T13:23:20.324343+0000","last_peered":"2024-03-30T13:23:20.324343+0000","last_clean":"2024-03-30T13:23:20.324343+0000","last_became_active":"2024-03-30T13:23:06.714191+0000","last_became_peered":"2024-03-30T13:23:06.714191+0000","last_unstale":"2024-03-30T13:23:20.324343+0000","last_undegraded":"2024-03-30T13:23:20.324343+0000","last_fullsized":"2024-03-30T13:23:20.324343+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:23:20.324157+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:23:20.324157+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:35.440933+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":395,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-03-30T13:23:08.575760+0000","last_change":"2024-03-30T13:22:27.910195+0000","last_active":"2024-03-30T13:23:08.575760+0000","last_peered":"2024-03-30T13:23:08.575760+0000","last_clean":"2024-03-30T13:23:08.575760+0000","last_became_active":"2024-03-30T13:21:56.541153+0000","last_became_peered":"2024-03-30T13:21:56.541153+0000","last_unstale":"2024-03-30T13:23:08.575760+0000","last_undegraded":"2024-03-30T13:23:08.575760+0000","last_fullsized":"2024-03-30T13:23:08.575760+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:22:27.909908+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:21:26.828701+0000","last_clean_scrub_stamp":"2024-03-30T13:22:27.909908+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:35.779509+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":575,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-03-30T13:23:23.425467+0000","last_change":"2024-03-30T13:23:23.425617+0000","last_active":"2024-03-30T13:23:23.425467+0000","last_peered":"2024-03-30T13:23:23.425467+0000","last_clean":"2024-03-30T13:23:23.425467+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:23:23.425467+0000","last_undegraded":"2024-03-30T13:23:23.425467+0000","last_fullsized":"2024-03-30T13:23:23.425467+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:23.425236+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:23:23.425236+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:24.000232+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1636,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-03-30T13:23:07.708767+0000","last_change":"2024-03-30T13:22:47.492011+0000","last_active":"2024-03-30T13:23:07.708767+0000","last_peered":"2024-03-30T13:23:07.708767+0000","last_clean":"2024-03-30T13:23:07.708767+0000","last_became_active":"2024-03-30T13:22:47.491313+0000","last_became_peered":"2024-03-30T13:22:47.491313+0000","last_unstale":"2024-03-30T13:23:07.708767+0000","last_undegraded":"2024-03-30T13:23:07.708767+0000","last_fullsized":"2024-03-30T13:23:07.708767+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:22:29.788082+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:22:29.788082+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:39.706081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":437,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-03-30T13:23:22.286263+0000","last_change":"2024-03-30T13:23:22.286357+0000","last_active":"2024-03-30T13:23:22.286263+0000","last_peered":"2024-03-30T13:23:22.286263+0000","last_clean":"2024-03-30T13:23:22.286263+0000","last_became_active":"2024-03-30T13:20:14.421760+0000","last_became_peered":"2024-03-30T13:20:14.421760+0000","last_unstale":"2024-03-30T13:23:22.286263+0000","last_undegraded":"2024-03-30T13:23:22.286263+0000","last_fullsized":"2024-03-30T13:23:22.286263+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:23:22.286071+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:23:22.286071+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:25.807487+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":21,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-03-30T13:23:07.709103+0000","last_change":"2024-03-30T13:23:06.770455+0000","last_active":"2024-03-30T13:23:07.709103+0000","last_peered":"2024-03-30T13:23:07.709103+0000","last_clean":"2024-03-30T13:23:07.709103+0000","last_became_active":"2024-03-30T13:23:06.712341+0000","last_became_peered":"2024-03-30T13:23:06.712341+0000","last_unstale":"2024-03-30T13:23:07.709103+0000","last_undegraded":"2024-03-30T13:23:07.709103+0000","last_fullsized":"2024-03-30T13:23:07.709103+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:23:02.469679+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:23:02.469679+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:19.428398+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1739,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-03-30T13:23:07.708553+0000","last_change":"2024-03-30T13:22:47.491942+0000","last_active":"2024-03-30T13:23:07.708553+0000","last_peered":"2024-03-30T13:23:07.708553+0000","last_clean":"2024-03-30T13:23:07.708553+0000","last_became_active":"2024-03-30T13:22:47.491356+0000","last_became_peered":"2024-03-30T13:22:47.491356+0000","last_unstale":"2024-03-30T13:23:07.708553+0000","last_undegraded":"2024-03-30T13:23:07.708553+0000","last_fullsized":"2024-03-30T13:23:07.708553+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:22:16.776051+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:16:45.572362+0000","last_clean_scrub_stamp":"2024-03-30T13:22:16.776051+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:25.260656+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1467,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-03-30T13:23:10.054018+0000","last_change":"2024-03-30T13:23:10.054085+0000","last_active":"2024-03-30T13:23:10.054018+0000","last_peered":"2024-03-30T13:23:10.054018+0000","last_clean":"2024-03-30T13:23:10.054018+0000","last_became_active":"2024-03-30T13:21:56.531330+0000","last_became_peered":"2024-03-30T13:21:56.531330+0000","last_unstale":"2024-03-30T13:23:10.054018+0000","last_undegraded":"2024-03-30T13:23:10.054018+0000","last_fullsized":"2024-03-30T13:23:10.054018+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:10.053837+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:23:10.053837+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:22.003357+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1718,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-03-30T13:23:19.612146+0000","last_change":"2024-03-30T13:23:19.612300+0000","last_active":"2024-03-30T13:23:19.612146+0000","last_peered":"2024-03-30T13:23:19.612146+0000","last_clean":"2024-03-30T13:23:19.612146+0000","last_became_active":"2024-03-30T13:21:56.537406+0000","last_became_peered":"2024-03-30T13:21:56.537406+0000","last_unstale":"2024-03-30T13:23:19.612146+0000","last_undegraded":"2024-03-30T13:23:19.612146+0000","last_fullsized":"2024-03-30T13:23:19.612146+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:23:19.611935+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:23:19.611935+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:48.677040+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1805,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-03-30T13:23:22.573618+0000","last_change":"2024-03-30T13:23:22.573707+0000","last_active":"2024-03-30T13:23:22.573618+0000","last_peered":"2024-03-30T13:23:22.573618+0000","last_clean":"2024-03-30T13:23:22.573618+0000","last_became_active":"2024-03-30T13:22:47.488417+0000","last_became_peered":"2024-03-30T13:22:47.488417+0000","last_unstale":"2024-03-30T13:23:22.573618+0000","last_undegraded":"2024-03-30T13:23:22.573618+0000","last_fullsized":"2024-03-30T13:23:22.573618+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:23:22.573400+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:43.538364+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":630,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-03-30T13:23:08.576624+0000","last_change":"2024-03-30T13:22:53.439552+0000","last_active":"2024-03-30T13:23:08.576624+0000","last_peered":"2024-03-30T13:23:08.576624+0000","last_clean":"2024-03-30T13:23:08.576624+0000","last_became_active":"2024-03-30T13:22:47.493033+0000","last_became_peered":"2024-03-30T13:22:47.493033+0000","last_unstale":"2024-03-30T13:23:08.576624+0000","last_undegraded":"2024-03-30T13:23:08.576624+0000","last_fullsized":"2024-03-30T13:23:08.576624+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:22:53.439303+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:22:53.439303+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:10.009027+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18319366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":398,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-03-30T13:23:08.575740+0000","last_change":"2024-03-30T13:22:47.490299+0000","last_active":"2024-03-30T13:23:08.575740+0000","last_peered":"2024-03-30T13:23:08.575740+0000","last_clean":"2024-03-30T13:23:08.575740+0000","last_became_active":"2024-03-30T13:22:47.489008+0000","last_became_peered":"2024-03-30T13:22:47.489008+0000","last_unstale":"2024-03-30T13:23:08.575740+0000","last_undegraded":"2024-03-30T13:23:08.575740+0000","last_fullsized":"2024-03-30T13:23:08.575740+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:22:38.916929+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:22:38.916929+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:51.005597+0000","scrub_duration":883,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4974806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":607,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-03-30T13:23:08.576647+0000","last_change":"2024-03-30T13:22:18.351128+0000","last_active":"2024-03-30T13:23:08.576647+0000","last_peered":"2024-03-30T13:23:08.576647+0000","last_clean":"2024-03-30T13:23:08.576647+0000","last_became_active":"2024-03-30T13:21:56.534725+0000","last_became_peered":"2024-03-30T13:21:56.534725+0000","last_unstale":"2024-03-30T13:23:08.576647+0000","last_undegraded":"2024-03-30T13:23:08.576647+0000","last_fullsized":"2024-03-30T13:23:08.576647+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:22:18.350810+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:22:18.350810+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:23:29.264706+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":27,"reported_epoch":766,"state":"active+clean","last_fresh":"2024-03-30T13:23:25.786075+0000","last_change":"2024-03-30T13:23:25.786146+0000","last_active":"2024-03-30T13:23:25.786075+0000","last_peered":"2024-03-30T13:23:25.786075+0000","last_clean":"2024-03-30T13:23:25.786075+0000","last_became_active":"2024-03-30T13:23:06.710101+0000","last_became_peered":"2024-03-30T13:23:06.710101+0000","last_unstale":"2024-03-30T13:23:25.786075+0000","last_undegraded":"2024-03-30T13:23:25.786075+0000","last_fullsized":"2024-03-30T13:23:25.786075+0000","mapping_epoch":764,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":765,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:23:25.785886+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:23:25.785886+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:37.915232+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5226911,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":270101667,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102486016,"data_stored":102486016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387732,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59340,"kb_used_data":22148,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94312500,"statfs":{"total":96636764160,"available":96576000000,"internally_reserved":0,"allocated":22679552,"data_stored":18738963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157548,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64352,"kb_used_data":26712,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94307488,"statfs":{"total":96636764160,"available":96570867712,"internally_reserved":0,"allocated":27353088,"data_stored":23514314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":677,"seq":2907692859433,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61988,"kb_used_data":20188,"kb_used_omap":13,"kb_used_meta":41778,"kb_avail":94309852,"statfs":{"total":96636764160,"available":96573288448,"internally_reserved":0,"allocated":20672512,"data_stored":16933668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14287,"internal_metadata":42780721},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509779,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52168,"kb_used_data":16832,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94319672,"statfs":{"total":96636764160,"available":96583344128,"internally_reserved":0,"allocated":17235968,"data_stored":13397194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509780,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55696,"kb_used_data":20360,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94316144,"statfs":{"total":96636764160,"available":96579731456,"internally_reserved":0,"allocated":20848640,"data_stored":17009866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742452,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57668,"kb_used_data":17084,"kb_used_omap":15,"kb_used_meta":40560,"kb_avail":94314172,"statfs":{"total":96636764160,"available":96577712128,"internally_reserved":0,"allocated":17494016,"data_stored":13553427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":41533891},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570122,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56832,"kb_used_data":19720,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94315008,"statfs":{"total":96636764160,"available":96578568192,"internally_reserved":0,"allocated":20193280,"data_stored":16152443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10211328,"data_stored":10211328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":13832192,"data_stored":13832192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:23:31.352 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:23:31.352 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:23:31.353 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-03-30T13:23:31.353 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-30T13:23:31.353 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-30T13:23:31.355 INFO:tasks.ceph.osd.1:Started 2024-03-30T13:23:31.355 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-30T13:23:31.419 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:23:31.418+0000 7fbca24c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:31.426 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:23:31.422+0000 7fbca24c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:31.426 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:23:31.422+0000 7fbca24c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:31.476 INFO:teuthology.orchestra.run.smithi067.stderr:no valid command found; 10 closest matches: 2024-03-30T13:23:31.476 INFO:teuthology.orchestra.run.smithi067.stderr:0 2024-03-30T13:23:31.476 INFO:teuthology.orchestra.run.smithi067.stderr:1 2024-03-30T13:23:31.477 INFO:teuthology.orchestra.run.smithi067.stderr:2 2024-03-30T13:23:31.477 INFO:teuthology.orchestra.run.smithi067.stderr:abort 2024-03-30T13:23:31.477 INFO:teuthology.orchestra.run.smithi067.stderr:assert 2024-03-30T13:23:31.477 INFO:teuthology.orchestra.run.smithi067.stderr:config diff 2024-03-30T13:23:31.477 INFO:teuthology.orchestra.run.smithi067.stderr:config diff get 2024-03-30T13:23:31.477 INFO:teuthology.orchestra.run.smithi067.stderr:config get 2024-03-30T13:23:31.477 INFO:teuthology.orchestra.run.smithi067.stderr:config help [] 2024-03-30T13:23:31.477 INFO:teuthology.orchestra.run.smithi067.stderr:config set ... 2024-03-30T13:23:31.477 INFO:teuthology.orchestra.run.smithi067.stderr:admin_socket: invalid command 2024-03-30T13:23:31.479 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:23:31.479 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-30T13:23:31.960 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:23:31.958+0000 7fbca24c97c0 -1 Falling back to public interface 2024-03-30T13:23:35.763 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:23:35.758+0000 7fbca24c97c0 -1 osd.1 763 log_to_monitors true 2024-03-30T13:23:36.479 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:23:36.478+0000 7fbc96296640 -1 osd.1 763 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:23:36.481 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-30T13:23:36.615 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-30T13:23:36.615 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-30T13:23:36.615 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-03-30T13:23:36.615 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-30T13:23:36.629 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:23:36.629 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-03-30T13:23:36.630 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 1 2024-03-30T13:23:36.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:36.746+0000 7fbb40908640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:36.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:36.762+0000 7fbb40908640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:36.945 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:23:36.942+0000 7fbb38f08640 -1 mon.a@0(leader).osd e768 definitely_dead 0 2024-03-30T13:23:37.459 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.1. 2024-03-30T13:23:37.576 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-03-30T13:23:37.576 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:23:37.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:37.702+0000 7f31a025d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:37.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:37.718+0000 7f31a025d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:38.467 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:23:38.495 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:23:38.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:38.606+0000 7ff575002640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:38.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:38.622+0000 7ff575002640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:39.469 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:23:39.498 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:23:39.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:39.618+0000 7f796051d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:39.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:39.638+0000 7f796051d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:40.489 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:23:40.515 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:23:40.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:40.634+0000 7f7445067640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:40.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:40.650+0000 7f7445067640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:41.498 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:23:41.524 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:23:41.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:41.642+0000 7f533f188640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:41.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:41.654+0000 7f533f188640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:42.514 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:23:42.541 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:23:42.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:42.658+0000 7f0d49006640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:42.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:42.674+0000 7f0d49006640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:43.517 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:23:43.545 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:23:43.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:43.666+0000 7f37e7e75640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:43.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:43.682+0000 7f37e7e75640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:44.528 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:23:44.551 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:23:44.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:44.662+0000 7f8936736640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:44.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:44.678+0000 7f8936736640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:45.537 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:23:45.565 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:23:45.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:45.682+0000 7f386ec80640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:45.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:45.702+0000 7f386ec80640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:46.552 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:23:46.576 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:23:46.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:46.706+0000 7f3db33c2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:46.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:46.722+0000 7f3db33c2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:47.556 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:23:47.583 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:23:47.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:47.702+0000 7f86e7b02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:47.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:47.714+0000 7f86e7b02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:48.573 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:23:48.596 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:23:48.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:48.718+0000 7ff07b0f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:48.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:48.734+0000 7ff07b0f5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:49.583 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:23:49.607 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:23:49.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:49.726+0000 7efd82c69640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:49.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:49.742+0000 7efd82c69640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:50.595 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:23:50.619 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:23:50.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:50.742+0000 7f62e4174640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:50.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:50.758+0000 7f62e4174640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:51.598 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:23:51.624 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:23:51.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:51.738+0000 7fa249442640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:51.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:51.754+0000 7fa249442640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:52.622 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:23:52.644 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:23:52.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:52.762+0000 7f7bff335640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:52.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:52.782+0000 7f7bff335640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:53.624 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:23:53.653 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:23:53.654 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:23:53.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:53.774+0000 7f59e2e38640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:53.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:53.790+0000 7f59e2e38640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:53.962 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:23:53.962 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:23:53.980 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1628,"stamp":"2024-03-30T13:23:53.629758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":264874756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":447812,"kb_used_data":145464,"kb_used_omap":70,"kb_used_meta":302137,"kb_avail":754526908,"statfs":{"total":773094113280,"available":772635553792,"internally_reserved":0,"allocated":148955136,"data_stored":120419293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":72445,"internal_metadata":309388547},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.049442"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1720,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-03-30T13:23:50.506781+0000","last_change":"2024-03-30T13:22:56.778810+0000","last_active":"2024-03-30T13:23:50.506781+0000","last_peered":"2024-03-30T13:23:50.506781+0000","last_clean":"2024-03-30T13:23:50.506781+0000","last_became_active":"2024-03-30T13:21:56.537232+0000","last_became_peered":"2024-03-30T13:21:56.537232+0000","last_unstale":"2024-03-30T13:23:50.506781+0000","last_undegraded":"2024-03-30T13:23:50.506781+0000","last_fullsized":"2024-03-30T13:23:50.506781+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:22:56.778530+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:22:56.778530+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:21.357262+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1679,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-03-30T13:23:53.378519+0000","last_change":"2024-03-30T13:23:30.947379+0000","last_active":"2024-03-30T13:23:53.378519+0000","last_peered":"2024-03-30T13:23:53.378519+0000","last_clean":"2024-03-30T13:23:53.378519+0000","last_became_active":"2024-03-30T13:22:47.491927+0000","last_became_peered":"2024-03-30T13:22:47.491927+0000","last_unstale":"2024-03-30T13:23:53.378519+0000","last_undegraded":"2024-03-30T13:23:53.378519+0000","last_fullsized":"2024-03-30T13:23:53.378519+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:30.947058+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:23:30.947058+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:31.095025+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":441,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:50.949900+0000","last_change":"2024-03-30T13:23:38.489520+0000","last_active":"2024-03-30T13:23:50.949900+0000","last_peered":"2024-03-30T13:23:50.949900+0000","last_clean":"2024-03-30T13:23:50.949900+0000","last_became_active":"2024-03-30T13:23:38.488514+0000","last_became_peered":"2024-03-30T13:23:38.488514+0000","last_unstale":"2024-03-30T13:23:50.949900+0000","last_undegraded":"2024-03-30T13:23:50.949900+0000","last_fullsized":"2024-03-30T13:23:50.949900+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:23:20.324157+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:23:20.324157+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:48.033710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":434,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:50.949725+0000","last_change":"2024-03-30T13:23:36.180822+0000","last_active":"2024-03-30T13:23:50.949725+0000","last_peered":"2024-03-30T13:23:50.949725+0000","last_clean":"2024-03-30T13:23:50.949725+0000","last_became_active":"2024-03-30T13:21:56.541153+0000","last_became_peered":"2024-03-30T13:21:56.541153+0000","last_unstale":"2024-03-30T13:23:50.949725+0000","last_undegraded":"2024-03-30T13:23:50.949725+0000","last_fullsized":"2024-03-30T13:23:50.949725+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:23:36.180557+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:52.909200+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":607,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:51.024749+0000","last_change":"2024-03-30T13:23:23.425617+0000","last_active":"2024-03-30T13:23:51.024749+0000","last_peered":"2024-03-30T13:23:51.024749+0000","last_clean":"2024-03-30T13:23:51.024749+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:23:51.024749+0000","last_undegraded":"2024-03-30T13:23:51.024749+0000","last_fullsized":"2024-03-30T13:23:51.024749+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:23.425236+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:23:23.425236+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:24.000232+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1676,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:50.595955+0000","last_change":"2024-03-30T13:23:40.597999+0000","last_active":"2024-03-30T13:23:50.595955+0000","last_peered":"2024-03-30T13:23:50.595955+0000","last_clean":"2024-03-30T13:23:50.595955+0000","last_became_active":"2024-03-30T13:22:47.491313+0000","last_became_peered":"2024-03-30T13:22:47.491313+0000","last_unstale":"2024-03-30T13:23:50.595955+0000","last_undegraded":"2024-03-30T13:23:50.595955+0000","last_fullsized":"2024-03-30T13:23:50.595955+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:23:40.597839+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:23:40.597839+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:00.377458+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":469,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:50.949760+0000","last_change":"2024-03-30T13:23:22.286357+0000","last_active":"2024-03-30T13:23:50.949760+0000","last_peered":"2024-03-30T13:23:50.949760+0000","last_clean":"2024-03-30T13:23:50.949760+0000","last_became_active":"2024-03-30T13:20:14.421760+0000","last_became_peered":"2024-03-30T13:20:14.421760+0000","last_unstale":"2024-03-30T13:23:50.949760+0000","last_undegraded":"2024-03-30T13:23:50.949760+0000","last_fullsized":"2024-03-30T13:23:50.949760+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:23:22.286071+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:23:22.286071+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:25.807487+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1039,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-03-30T13:23:52.172637+0000","last_change":"2024-03-30T13:23:38.515178+0000","last_active":"2024-03-30T13:23:52.172637+0000","last_peered":"2024-03-30T13:23:52.172637+0000","last_clean":"2024-03-30T13:23:52.172637+0000","last_became_active":"2024-03-30T13:23:38.514736+0000","last_became_peered":"2024-03-30T13:23:38.514736+0000","last_unstale":"2024-03-30T13:23:52.172637+0000","last_undegraded":"2024-03-30T13:23:52.172637+0000","last_fullsized":"2024-03-30T13:23:52.172637+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:23:02.469679+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:23:02.469679+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:23.725884+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1779,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:50.595784+0000","last_change":"2024-03-30T13:23:27.681044+0000","last_active":"2024-03-30T13:23:50.595784+0000","last_peered":"2024-03-30T13:23:50.595784+0000","last_clean":"2024-03-30T13:23:50.595784+0000","last_became_active":"2024-03-30T13:22:47.491356+0000","last_became_peered":"2024-03-30T13:22:47.491356+0000","last_unstale":"2024-03-30T13:23:50.595784+0000","last_undegraded":"2024-03-30T13:23:50.595784+0000","last_fullsized":"2024-03-30T13:23:50.595784+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:23:27.680784+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:45.246430+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1501,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-03-30T13:23:53.378141+0000","last_change":"2024-03-30T13:23:10.054085+0000","last_active":"2024-03-30T13:23:53.378141+0000","last_peered":"2024-03-30T13:23:53.378141+0000","last_clean":"2024-03-30T13:23:53.378141+0000","last_became_active":"2024-03-30T13:21:56.531330+0000","last_became_peered":"2024-03-30T13:21:56.531330+0000","last_unstale":"2024-03-30T13:23:53.378141+0000","last_undegraded":"2024-03-30T13:23:53.378141+0000","last_fullsized":"2024-03-30T13:23:53.378141+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:10.053837+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:23:10.053837+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:22.003357+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1747,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-03-30T13:23:50.506650+0000","last_change":"2024-03-30T13:23:19.612300+0000","last_active":"2024-03-30T13:23:50.506650+0000","last_peered":"2024-03-30T13:23:50.506650+0000","last_clean":"2024-03-30T13:23:50.506650+0000","last_became_active":"2024-03-30T13:21:56.537406+0000","last_became_peered":"2024-03-30T13:21:56.537406+0000","last_unstale":"2024-03-30T13:23:50.506650+0000","last_undegraded":"2024-03-30T13:23:50.506650+0000","last_fullsized":"2024-03-30T13:23:50.506650+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:23:19.611935+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:23:19.611935+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:48.677040+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1834,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-03-30T13:23:50.506651+0000","last_change":"2024-03-30T13:23:22.573707+0000","last_active":"2024-03-30T13:23:50.506651+0000","last_peered":"2024-03-30T13:23:50.506651+0000","last_clean":"2024-03-30T13:23:50.506651+0000","last_became_active":"2024-03-30T13:22:47.488417+0000","last_became_peered":"2024-03-30T13:22:47.488417+0000","last_unstale":"2024-03-30T13:23:50.506651+0000","last_undegraded":"2024-03-30T13:23:50.506651+0000","last_fullsized":"2024-03-30T13:23:50.506651+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:23:22.573400+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:43.538364+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":662,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:51.024723+0000","last_change":"2024-03-30T13:22:53.439552+0000","last_active":"2024-03-30T13:23:51.024723+0000","last_peered":"2024-03-30T13:23:51.024723+0000","last_clean":"2024-03-30T13:23:51.024723+0000","last_became_active":"2024-03-30T13:22:47.493033+0000","last_became_peered":"2024-03-30T13:22:47.493033+0000","last_unstale":"2024-03-30T13:23:51.024723+0000","last_undegraded":"2024-03-30T13:23:51.024723+0000","last_fullsized":"2024-03-30T13:23:51.024723+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:22:53.439303+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:22:53.439303+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:10.009027+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18319366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":438,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:51.071558+0000","last_change":"2024-03-30T13:23:51.071627+0000","last_active":"2024-03-30T13:23:51.071558+0000","last_peered":"2024-03-30T13:23:51.071558+0000","last_clean":"2024-03-30T13:23:51.071558+0000","last_became_active":"2024-03-30T13:22:47.489008+0000","last_became_peered":"2024-03-30T13:22:47.489008+0000","last_unstale":"2024-03-30T13:23:51.071558+0000","last_undegraded":"2024-03-30T13:23:51.071558+0000","last_fullsized":"2024-03-30T13:23:51.071558+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:51.071390+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:23:51.071390+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:10.481199+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4974806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":647,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:51.024757+0000","last_change":"2024-03-30T13:23:32.252109+0000","last_active":"2024-03-30T13:23:51.024757+0000","last_peered":"2024-03-30T13:23:51.024757+0000","last_clean":"2024-03-30T13:23:51.024757+0000","last_became_active":"2024-03-30T13:21:56.534725+0000","last_became_peered":"2024-03-30T13:21:56.534725+0000","last_unstale":"2024-03-30T13:23:51.024757+0000","last_undegraded":"2024-03-30T13:23:51.024757+0000","last_fullsized":"2024-03-30T13:23:51.024757+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:23:32.251853+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:23:32.251853+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:40.482169+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":902,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-03-30T13:23:52.172702+0000","last_change":"2024-03-30T13:23:38.517308+0000","last_active":"2024-03-30T13:23:52.172702+0000","last_peered":"2024-03-30T13:23:52.172702+0000","last_clean":"2024-03-30T13:23:52.172702+0000","last_became_active":"2024-03-30T13:23:38.516830+0000","last_became_peered":"2024-03-30T13:23:38.516830+0000","last_unstale":"2024-03-30T13:23:52.172702+0000","last_undegraded":"2024-03-30T13:23:52.172702+0000","last_fullsized":"2024-03-30T13:23:52.172702+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:23:25.785886+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:23:25.785886+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:26.017159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":264874756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387737,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58660,"kb_used_data":21464,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94313180,"statfs":{"total":96636764160,"available":96576696320,"internally_reserved":0,"allocated":21979136,"data_stored":18421507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157553,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63880,"kb_used_data":26232,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94307960,"statfs":{"total":96636764160,"available":96571351040,"internally_reserved":0,"allocated":26861568,"data_stored":23290337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":677,"seq":2907692859438,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59748,"kb_used_data":17936,"kb_used_omap":13,"kb_used_meta":41778,"kb_avail":94312092,"statfs":{"total":96636764160,"available":96575582208,"internally_reserved":0,"allocated":18366464,"data_stored":14795233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14287,"internal_metadata":42780721},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509783,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48040,"kb_used_data":12692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323800,"statfs":{"total":96636764160,"available":96587571200,"internally_reserved":0,"allocated":12996608,"data_stored":9432178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":769,"seq":3302829850629,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49624,"kb_used_data":11980,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94322216,"statfs":{"total":96636764160,"available":96585949184,"internally_reserved":0,"allocated":12267520,"data_stored":8696289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10995,"internal_metadata":38524173},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509784,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55204,"kb_used_data":19856,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94316636,"statfs":{"total":96636764160,"available":96580235264,"internally_reserved":0,"allocated":20332544,"data_stored":16774915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742457,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57020,"kb_used_data":16424,"kb_used_omap":15,"kb_used_meta":40560,"kb_avail":94314820,"statfs":{"total":96636764160,"available":96578375680,"internally_reserved":0,"allocated":16818176,"data_stored":13246945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":41533891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570127,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55636,"kb_used_data":18880,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94316204,"statfs":{"total":96636764160,"available":96579792896,"internally_reserved":0,"allocated":19333120,"data_stored":15761889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:23:53.982 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:23:54.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:54.094+0000 7f725a5b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:54.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:54.106+0000 7f725a5b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:54.276 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:23:54.277 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:23:54.293 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1628,"stamp":"2024-03-30T13:23:53.629758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":264874756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":447812,"kb_used_data":145464,"kb_used_omap":70,"kb_used_meta":302137,"kb_avail":754526908,"statfs":{"total":773094113280,"available":772635553792,"internally_reserved":0,"allocated":148955136,"data_stored":120419293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":72445,"internal_metadata":309388547},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.049442"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1720,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-03-30T13:23:50.506781+0000","last_change":"2024-03-30T13:22:56.778810+0000","last_active":"2024-03-30T13:23:50.506781+0000","last_peered":"2024-03-30T13:23:50.506781+0000","last_clean":"2024-03-30T13:23:50.506781+0000","last_became_active":"2024-03-30T13:21:56.537232+0000","last_became_peered":"2024-03-30T13:21:56.537232+0000","last_unstale":"2024-03-30T13:23:50.506781+0000","last_undegraded":"2024-03-30T13:23:50.506781+0000","last_fullsized":"2024-03-30T13:23:50.506781+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:22:56.778530+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:22:56.778530+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:21.357262+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1679,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-03-30T13:23:53.378519+0000","last_change":"2024-03-30T13:23:30.947379+0000","last_active":"2024-03-30T13:23:53.378519+0000","last_peered":"2024-03-30T13:23:53.378519+0000","last_clean":"2024-03-30T13:23:53.378519+0000","last_became_active":"2024-03-30T13:22:47.491927+0000","last_became_peered":"2024-03-30T13:22:47.491927+0000","last_unstale":"2024-03-30T13:23:53.378519+0000","last_undegraded":"2024-03-30T13:23:53.378519+0000","last_fullsized":"2024-03-30T13:23:53.378519+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:30.947058+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:23:30.947058+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:31.095025+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":441,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:50.949900+0000","last_change":"2024-03-30T13:23:38.489520+0000","last_active":"2024-03-30T13:23:50.949900+0000","last_peered":"2024-03-30T13:23:50.949900+0000","last_clean":"2024-03-30T13:23:50.949900+0000","last_became_active":"2024-03-30T13:23:38.488514+0000","last_became_peered":"2024-03-30T13:23:38.488514+0000","last_unstale":"2024-03-30T13:23:50.949900+0000","last_undegraded":"2024-03-30T13:23:50.949900+0000","last_fullsized":"2024-03-30T13:23:50.949900+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:23:20.324157+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:23:20.324157+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:48.033710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":434,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:50.949725+0000","last_change":"2024-03-30T13:23:36.180822+0000","last_active":"2024-03-30T13:23:50.949725+0000","last_peered":"2024-03-30T13:23:50.949725+0000","last_clean":"2024-03-30T13:23:50.949725+0000","last_became_active":"2024-03-30T13:21:56.541153+0000","last_became_peered":"2024-03-30T13:21:56.541153+0000","last_unstale":"2024-03-30T13:23:50.949725+0000","last_undegraded":"2024-03-30T13:23:50.949725+0000","last_fullsized":"2024-03-30T13:23:50.949725+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:23:36.180557+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:52.909200+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":607,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:51.024749+0000","last_change":"2024-03-30T13:23:23.425617+0000","last_active":"2024-03-30T13:23:51.024749+0000","last_peered":"2024-03-30T13:23:51.024749+0000","last_clean":"2024-03-30T13:23:51.024749+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:23:51.024749+0000","last_undegraded":"2024-03-30T13:23:51.024749+0000","last_fullsized":"2024-03-30T13:23:51.024749+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:23.425236+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:23:23.425236+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:24.000232+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1676,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:50.595955+0000","last_change":"2024-03-30T13:23:40.597999+0000","last_active":"2024-03-30T13:23:50.595955+0000","last_peered":"2024-03-30T13:23:50.595955+0000","last_clean":"2024-03-30T13:23:50.595955+0000","last_became_active":"2024-03-30T13:22:47.491313+0000","last_became_peered":"2024-03-30T13:22:47.491313+0000","last_unstale":"2024-03-30T13:23:50.595955+0000","last_undegraded":"2024-03-30T13:23:50.595955+0000","last_fullsized":"2024-03-30T13:23:50.595955+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:23:40.597839+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:23:40.597839+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:00.377458+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":469,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:50.949760+0000","last_change":"2024-03-30T13:23:22.286357+0000","last_active":"2024-03-30T13:23:50.949760+0000","last_peered":"2024-03-30T13:23:50.949760+0000","last_clean":"2024-03-30T13:23:50.949760+0000","last_became_active":"2024-03-30T13:20:14.421760+0000","last_became_peered":"2024-03-30T13:20:14.421760+0000","last_unstale":"2024-03-30T13:23:50.949760+0000","last_undegraded":"2024-03-30T13:23:50.949760+0000","last_fullsized":"2024-03-30T13:23:50.949760+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:23:22.286071+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:23:22.286071+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:25.807487+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1039,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-03-30T13:23:52.172637+0000","last_change":"2024-03-30T13:23:38.515178+0000","last_active":"2024-03-30T13:23:52.172637+0000","last_peered":"2024-03-30T13:23:52.172637+0000","last_clean":"2024-03-30T13:23:52.172637+0000","last_became_active":"2024-03-30T13:23:38.514736+0000","last_became_peered":"2024-03-30T13:23:38.514736+0000","last_unstale":"2024-03-30T13:23:52.172637+0000","last_undegraded":"2024-03-30T13:23:52.172637+0000","last_fullsized":"2024-03-30T13:23:52.172637+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:23:02.469679+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:23:02.469679+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:23.725884+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1779,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:50.595784+0000","last_change":"2024-03-30T13:23:27.681044+0000","last_active":"2024-03-30T13:23:50.595784+0000","last_peered":"2024-03-30T13:23:50.595784+0000","last_clean":"2024-03-30T13:23:50.595784+0000","last_became_active":"2024-03-30T13:22:47.491356+0000","last_became_peered":"2024-03-30T13:22:47.491356+0000","last_unstale":"2024-03-30T13:23:50.595784+0000","last_undegraded":"2024-03-30T13:23:50.595784+0000","last_fullsized":"2024-03-30T13:23:50.595784+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:23:27.680784+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:45.246430+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1501,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-03-30T13:23:53.378141+0000","last_change":"2024-03-30T13:23:10.054085+0000","last_active":"2024-03-30T13:23:53.378141+0000","last_peered":"2024-03-30T13:23:53.378141+0000","last_clean":"2024-03-30T13:23:53.378141+0000","last_became_active":"2024-03-30T13:21:56.531330+0000","last_became_peered":"2024-03-30T13:21:56.531330+0000","last_unstale":"2024-03-30T13:23:53.378141+0000","last_undegraded":"2024-03-30T13:23:53.378141+0000","last_fullsized":"2024-03-30T13:23:53.378141+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:10.053837+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:23:10.053837+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:22.003357+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1747,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-03-30T13:23:50.506650+0000","last_change":"2024-03-30T13:23:19.612300+0000","last_active":"2024-03-30T13:23:50.506650+0000","last_peered":"2024-03-30T13:23:50.506650+0000","last_clean":"2024-03-30T13:23:50.506650+0000","last_became_active":"2024-03-30T13:21:56.537406+0000","last_became_peered":"2024-03-30T13:21:56.537406+0000","last_unstale":"2024-03-30T13:23:50.506650+0000","last_undegraded":"2024-03-30T13:23:50.506650+0000","last_fullsized":"2024-03-30T13:23:50.506650+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:23:19.611935+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:23:19.611935+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:48.677040+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1834,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-03-30T13:23:50.506651+0000","last_change":"2024-03-30T13:23:22.573707+0000","last_active":"2024-03-30T13:23:50.506651+0000","last_peered":"2024-03-30T13:23:50.506651+0000","last_clean":"2024-03-30T13:23:50.506651+0000","last_became_active":"2024-03-30T13:22:47.488417+0000","last_became_peered":"2024-03-30T13:22:47.488417+0000","last_unstale":"2024-03-30T13:23:50.506651+0000","last_undegraded":"2024-03-30T13:23:50.506651+0000","last_fullsized":"2024-03-30T13:23:50.506651+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:23:22.573400+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:43.538364+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":662,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:51.024723+0000","last_change":"2024-03-30T13:22:53.439552+0000","last_active":"2024-03-30T13:23:51.024723+0000","last_peered":"2024-03-30T13:23:51.024723+0000","last_clean":"2024-03-30T13:23:51.024723+0000","last_became_active":"2024-03-30T13:22:47.493033+0000","last_became_peered":"2024-03-30T13:22:47.493033+0000","last_unstale":"2024-03-30T13:23:51.024723+0000","last_undegraded":"2024-03-30T13:23:51.024723+0000","last_fullsized":"2024-03-30T13:23:51.024723+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:22:53.439303+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:22:53.439303+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:10.009027+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18319366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":438,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:51.071558+0000","last_change":"2024-03-30T13:23:51.071627+0000","last_active":"2024-03-30T13:23:51.071558+0000","last_peered":"2024-03-30T13:23:51.071558+0000","last_clean":"2024-03-30T13:23:51.071558+0000","last_became_active":"2024-03-30T13:22:47.489008+0000","last_became_peered":"2024-03-30T13:22:47.489008+0000","last_unstale":"2024-03-30T13:23:51.071558+0000","last_undegraded":"2024-03-30T13:23:51.071558+0000","last_fullsized":"2024-03-30T13:23:51.071558+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:51.071390+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:23:51.071390+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:10.481199+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4974806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":647,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:51.024757+0000","last_change":"2024-03-30T13:23:32.252109+0000","last_active":"2024-03-30T13:23:51.024757+0000","last_peered":"2024-03-30T13:23:51.024757+0000","last_clean":"2024-03-30T13:23:51.024757+0000","last_became_active":"2024-03-30T13:21:56.534725+0000","last_became_peered":"2024-03-30T13:21:56.534725+0000","last_unstale":"2024-03-30T13:23:51.024757+0000","last_undegraded":"2024-03-30T13:23:51.024757+0000","last_fullsized":"2024-03-30T13:23:51.024757+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:23:32.251853+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:23:32.251853+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:40.482169+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":902,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-03-30T13:23:52.172702+0000","last_change":"2024-03-30T13:23:38.517308+0000","last_active":"2024-03-30T13:23:52.172702+0000","last_peered":"2024-03-30T13:23:52.172702+0000","last_clean":"2024-03-30T13:23:52.172702+0000","last_became_active":"2024-03-30T13:23:38.516830+0000","last_became_peered":"2024-03-30T13:23:38.516830+0000","last_unstale":"2024-03-30T13:23:52.172702+0000","last_undegraded":"2024-03-30T13:23:52.172702+0000","last_fullsized":"2024-03-30T13:23:52.172702+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:23:25.785886+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:23:25.785886+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:26.017159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":264874756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387737,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58660,"kb_used_data":21464,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94313180,"statfs":{"total":96636764160,"available":96576696320,"internally_reserved":0,"allocated":21979136,"data_stored":18421507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157553,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63880,"kb_used_data":26232,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94307960,"statfs":{"total":96636764160,"available":96571351040,"internally_reserved":0,"allocated":26861568,"data_stored":23290337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":677,"seq":2907692859438,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59748,"kb_used_data":17936,"kb_used_omap":13,"kb_used_meta":41778,"kb_avail":94312092,"statfs":{"total":96636764160,"available":96575582208,"internally_reserved":0,"allocated":18366464,"data_stored":14795233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14287,"internal_metadata":42780721},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509783,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48040,"kb_used_data":12692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323800,"statfs":{"total":96636764160,"available":96587571200,"internally_reserved":0,"allocated":12996608,"data_stored":9432178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":769,"seq":3302829850629,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49624,"kb_used_data":11980,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94322216,"statfs":{"total":96636764160,"available":96585949184,"internally_reserved":0,"allocated":12267520,"data_stored":8696289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10995,"internal_metadata":38524173},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509784,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55204,"kb_used_data":19856,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94316636,"statfs":{"total":96636764160,"available":96580235264,"internally_reserved":0,"allocated":20332544,"data_stored":16774915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742457,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57020,"kb_used_data":16424,"kb_used_omap":15,"kb_used_meta":40560,"kb_avail":94314820,"statfs":{"total":96636764160,"available":96578375680,"internally_reserved":0,"allocated":16818176,"data_stored":13246945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":41533891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570127,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55636,"kb_used_data":18880,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94316204,"statfs":{"total":96636764160,"available":96579792896,"internally_reserved":0,"allocated":19333120,"data_stored":15761889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:23:54.294 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:23:54.294 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:23:54.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:54.402+0000 7fa951220640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:54.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:54.414+0000 7fa951220640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:54.585 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:23:54.585 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":785,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:23:53.625305+0000","last_up_change":"2024-03-30T13:23:37.440586+0000","last_in_change":"2024-03-30T13:23:37.440586+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":38,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":739,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":581,"last_clean_end":763,"up_from":769,"up_thru":769,"down_at":764,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4049884108},{"type":"v1","addr":"172.21.15.67:6818","nonce":4049884108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4049884108},{"type":"v1","addr":"172.21.15.67:6821","nonce":4049884108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":4049884108},{"type":"v1","addr":"172.21.15.67:6828","nonce":4049884108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":4049884108},{"type":"v1","addr":"172.21.15.67:6825","nonce":4049884108}]},"public_addr":"172.21.15.67:6818/4049884108","cluster_addr":"172.21.15.67:6821/4049884108","heartbeat_back_addr":"172.21.15.67:6828/4049884108","heartbeat_front_addr":"172.21.15.67:6825/4049884108","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":764,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":699,"last_clean_end":738,"up_from":747,"up_thru":747,"down_at":739,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6801","nonce":3161027348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6803","nonce":3161027348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6807","nonce":3161027348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6805","nonce":3161027348}]},"public_addr":"172.21.15.67:6801/3161027348","cluster_addr":"172.21.15.67:6803/3161027348","heartbeat_back_addr":"172.21.15.67:6807/3161027348","heartbeat_front_addr":"172.21.15.67:6805/3161027348","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":559,"last_clean_end":666,"up_from":677,"up_thru":764,"down_at":667,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1606011629},{"type":"v1","addr":"172.21.15.82:6803","nonce":1606011629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":1606011629},{"type":"v1","addr":"172.21.15.82:6807","nonce":1606011629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":1606011629},{"type":"v1","addr":"172.21.15.82:6815","nonce":1606011629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":1606011629},{"type":"v1","addr":"172.21.15.82:6811","nonce":1606011629}]},"public_addr":"172.21.15.82:6803/1606011629","cluster_addr":"172.21.15.82:6807/1606011629","heartbeat_back_addr":"172.21.15.82:6815/1606011629","heartbeat_front_addr":"172.21.15.82:6811/1606011629","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":488,"last_clean_end":506,"up_from":515,"up_thru":747,"down_at":507,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6817","nonce":292505453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6819","nonce":292505453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6823","nonce":292505453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6821","nonce":292505453}]},"public_addr":"172.21.15.82:6817/292505453","cluster_addr":"172.21.15.82:6819/292505453","heartbeat_back_addr":"172.21.15.82:6823/292505453","heartbeat_front_addr":"172.21.15.82:6821/292505453","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":629,"last_clean_end":715,"up_from":722,"up_thru":747,"down_at":716,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6801","nonce":326680821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6806","nonce":326680821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6814","nonce":326680821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6810","nonce":326680821}]},"public_addr":"172.21.15.82:6801/326680821","cluster_addr":"172.21.15.82:6806/326680821","heartbeat_back_addr":"172.21.15.82:6814/326680821","heartbeat_front_addr":"172.21.15.82:6810/326680821","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":604,"last_clean_end":645,"up_from":650,"up_thru":769,"down_at":646,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6825","nonce":1497758647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6827","nonce":1497758647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6831","nonce":1497758647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6829","nonce":1497758647}]},"public_addr":"172.21.15.82:6825/1497758647","cluster_addr":"172.21.15.82:6827/1497758647","heartbeat_back_addr":"172.21.15.82:6831/1497758647","heartbeat_front_addr":"172.21.15.82:6829/1497758647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:22:47.482020+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:23:44.725062+0000","dead_epoch":764},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:23:46.060234+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:22:45.755217+0000","dead_epoch":739},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:23:07.666246+0000","dead_epoch":667},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:22:43.071176+0000","dead_epoch":507},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:23:30.766774+0000","dead_epoch":716},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:22:51.504399+0000","dead_epoch":646}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:23:54.602 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:23:54.602 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:23:54.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:54.710+0000 7f18e92a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:54.726+0000 7f18e92a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:54.889 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:23:54.906 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:23:55.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:55.022+0000 7f5616dbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:55.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:55.038+0000 7f5616dbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:55.218 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:23:55.218 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:23:55.237 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:23:55.238 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:23:55.238 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:23:55.238 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:23:55.238 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 5, 7, 4, 6, 3, 1] 2024-03-30T13:23:55.238 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:23:55.238 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:23:55.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:55.354+0000 7fccbf948640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:55.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:55.366+0000 7fccbf948640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:55.531 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:23:55.532 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:23:55.548 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1629,"stamp":"2024-03-30T13:23:54.277606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":264874756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":447812,"kb_used_data":145464,"kb_used_omap":70,"kb_used_meta":302137,"kb_avail":754526908,"statfs":{"total":773094113280,"available":772635553792,"internally_reserved":0,"allocated":148955136,"data_stored":120419293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":72445,"internal_metadata":309388547},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.004260"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1720,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-03-30T13:23:50.506781+0000","last_change":"2024-03-30T13:22:56.778810+0000","last_active":"2024-03-30T13:23:50.506781+0000","last_peered":"2024-03-30T13:23:50.506781+0000","last_clean":"2024-03-30T13:23:50.506781+0000","last_became_active":"2024-03-30T13:21:56.537232+0000","last_became_peered":"2024-03-30T13:21:56.537232+0000","last_unstale":"2024-03-30T13:23:50.506781+0000","last_undegraded":"2024-03-30T13:23:50.506781+0000","last_fullsized":"2024-03-30T13:23:50.506781+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:22:56.778530+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:22:56.778530+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:21.357262+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1679,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-03-30T13:23:53.378519+0000","last_change":"2024-03-30T13:23:30.947379+0000","last_active":"2024-03-30T13:23:53.378519+0000","last_peered":"2024-03-30T13:23:53.378519+0000","last_clean":"2024-03-30T13:23:53.378519+0000","last_became_active":"2024-03-30T13:22:47.491927+0000","last_became_peered":"2024-03-30T13:22:47.491927+0000","last_unstale":"2024-03-30T13:23:53.378519+0000","last_undegraded":"2024-03-30T13:23:53.378519+0000","last_fullsized":"2024-03-30T13:23:53.378519+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:30.947058+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:23:30.947058+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:31.095025+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":441,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:50.949900+0000","last_change":"2024-03-30T13:23:38.489520+0000","last_active":"2024-03-30T13:23:50.949900+0000","last_peered":"2024-03-30T13:23:50.949900+0000","last_clean":"2024-03-30T13:23:50.949900+0000","last_became_active":"2024-03-30T13:23:38.488514+0000","last_became_peered":"2024-03-30T13:23:38.488514+0000","last_unstale":"2024-03-30T13:23:50.949900+0000","last_undegraded":"2024-03-30T13:23:50.949900+0000","last_fullsized":"2024-03-30T13:23:50.949900+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:23:20.324157+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:23:20.324157+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:48.033710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":434,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:50.949725+0000","last_change":"2024-03-30T13:23:36.180822+0000","last_active":"2024-03-30T13:23:50.949725+0000","last_peered":"2024-03-30T13:23:50.949725+0000","last_clean":"2024-03-30T13:23:50.949725+0000","last_became_active":"2024-03-30T13:21:56.541153+0000","last_became_peered":"2024-03-30T13:21:56.541153+0000","last_unstale":"2024-03-30T13:23:50.949725+0000","last_undegraded":"2024-03-30T13:23:50.949725+0000","last_fullsized":"2024-03-30T13:23:50.949725+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:23:36.180557+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:52.909200+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":607,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:51.024749+0000","last_change":"2024-03-30T13:23:23.425617+0000","last_active":"2024-03-30T13:23:51.024749+0000","last_peered":"2024-03-30T13:23:51.024749+0000","last_clean":"2024-03-30T13:23:51.024749+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:23:51.024749+0000","last_undegraded":"2024-03-30T13:23:51.024749+0000","last_fullsized":"2024-03-30T13:23:51.024749+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:23.425236+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:23:23.425236+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:24.000232+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1676,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:50.595955+0000","last_change":"2024-03-30T13:23:40.597999+0000","last_active":"2024-03-30T13:23:50.595955+0000","last_peered":"2024-03-30T13:23:50.595955+0000","last_clean":"2024-03-30T13:23:50.595955+0000","last_became_active":"2024-03-30T13:22:47.491313+0000","last_became_peered":"2024-03-30T13:22:47.491313+0000","last_unstale":"2024-03-30T13:23:50.595955+0000","last_undegraded":"2024-03-30T13:23:50.595955+0000","last_fullsized":"2024-03-30T13:23:50.595955+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:23:40.597839+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:23:40.597839+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:00.377458+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":469,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:50.949760+0000","last_change":"2024-03-30T13:23:22.286357+0000","last_active":"2024-03-30T13:23:50.949760+0000","last_peered":"2024-03-30T13:23:50.949760+0000","last_clean":"2024-03-30T13:23:50.949760+0000","last_became_active":"2024-03-30T13:20:14.421760+0000","last_became_peered":"2024-03-30T13:20:14.421760+0000","last_unstale":"2024-03-30T13:23:50.949760+0000","last_undegraded":"2024-03-30T13:23:50.949760+0000","last_fullsized":"2024-03-30T13:23:50.949760+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:23:22.286071+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:23:22.286071+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:25.807487+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1039,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-03-30T13:23:52.172637+0000","last_change":"2024-03-30T13:23:38.515178+0000","last_active":"2024-03-30T13:23:52.172637+0000","last_peered":"2024-03-30T13:23:52.172637+0000","last_clean":"2024-03-30T13:23:52.172637+0000","last_became_active":"2024-03-30T13:23:38.514736+0000","last_became_peered":"2024-03-30T13:23:38.514736+0000","last_unstale":"2024-03-30T13:23:52.172637+0000","last_undegraded":"2024-03-30T13:23:52.172637+0000","last_fullsized":"2024-03-30T13:23:52.172637+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:23:02.469679+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:23:02.469679+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:23.725884+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1779,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:50.595784+0000","last_change":"2024-03-30T13:23:27.681044+0000","last_active":"2024-03-30T13:23:50.595784+0000","last_peered":"2024-03-30T13:23:50.595784+0000","last_clean":"2024-03-30T13:23:50.595784+0000","last_became_active":"2024-03-30T13:22:47.491356+0000","last_became_peered":"2024-03-30T13:22:47.491356+0000","last_unstale":"2024-03-30T13:23:50.595784+0000","last_undegraded":"2024-03-30T13:23:50.595784+0000","last_fullsized":"2024-03-30T13:23:50.595784+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:23:27.680784+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:45.246430+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1501,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-03-30T13:23:53.378141+0000","last_change":"2024-03-30T13:23:10.054085+0000","last_active":"2024-03-30T13:23:53.378141+0000","last_peered":"2024-03-30T13:23:53.378141+0000","last_clean":"2024-03-30T13:23:53.378141+0000","last_became_active":"2024-03-30T13:21:56.531330+0000","last_became_peered":"2024-03-30T13:21:56.531330+0000","last_unstale":"2024-03-30T13:23:53.378141+0000","last_undegraded":"2024-03-30T13:23:53.378141+0000","last_fullsized":"2024-03-30T13:23:53.378141+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:10.053837+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:23:10.053837+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:22.003357+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1747,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-03-30T13:23:50.506650+0000","last_change":"2024-03-30T13:23:19.612300+0000","last_active":"2024-03-30T13:23:50.506650+0000","last_peered":"2024-03-30T13:23:50.506650+0000","last_clean":"2024-03-30T13:23:50.506650+0000","last_became_active":"2024-03-30T13:21:56.537406+0000","last_became_peered":"2024-03-30T13:21:56.537406+0000","last_unstale":"2024-03-30T13:23:50.506650+0000","last_undegraded":"2024-03-30T13:23:50.506650+0000","last_fullsized":"2024-03-30T13:23:50.506650+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:23:19.611935+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:23:19.611935+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:48.677040+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1834,"reported_epoch":781,"state":"active+clean","last_fresh":"2024-03-30T13:23:50.506651+0000","last_change":"2024-03-30T13:23:22.573707+0000","last_active":"2024-03-30T13:23:50.506651+0000","last_peered":"2024-03-30T13:23:50.506651+0000","last_clean":"2024-03-30T13:23:50.506651+0000","last_became_active":"2024-03-30T13:22:47.488417+0000","last_became_peered":"2024-03-30T13:22:47.488417+0000","last_unstale":"2024-03-30T13:23:50.506651+0000","last_undegraded":"2024-03-30T13:23:50.506651+0000","last_fullsized":"2024-03-30T13:23:50.506651+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:23:22.573400+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:43.538364+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":662,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:51.024723+0000","last_change":"2024-03-30T13:22:53.439552+0000","last_active":"2024-03-30T13:23:51.024723+0000","last_peered":"2024-03-30T13:23:51.024723+0000","last_clean":"2024-03-30T13:23:51.024723+0000","last_became_active":"2024-03-30T13:22:47.493033+0000","last_became_peered":"2024-03-30T13:22:47.493033+0000","last_unstale":"2024-03-30T13:23:51.024723+0000","last_undegraded":"2024-03-30T13:23:51.024723+0000","last_fullsized":"2024-03-30T13:23:51.024723+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:22:53.439303+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:22:53.439303+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:10.009027+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18319366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":438,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:51.071558+0000","last_change":"2024-03-30T13:23:51.071627+0000","last_active":"2024-03-30T13:23:51.071558+0000","last_peered":"2024-03-30T13:23:51.071558+0000","last_clean":"2024-03-30T13:23:51.071558+0000","last_became_active":"2024-03-30T13:22:47.489008+0000","last_became_peered":"2024-03-30T13:22:47.489008+0000","last_unstale":"2024-03-30T13:23:51.071558+0000","last_undegraded":"2024-03-30T13:23:51.071558+0000","last_fullsized":"2024-03-30T13:23:51.071558+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:51.071390+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:23:51.071390+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:10.481199+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4974806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":647,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:51.024757+0000","last_change":"2024-03-30T13:23:32.252109+0000","last_active":"2024-03-30T13:23:51.024757+0000","last_peered":"2024-03-30T13:23:51.024757+0000","last_clean":"2024-03-30T13:23:51.024757+0000","last_became_active":"2024-03-30T13:21:56.534725+0000","last_became_peered":"2024-03-30T13:21:56.534725+0000","last_unstale":"2024-03-30T13:23:51.024757+0000","last_undegraded":"2024-03-30T13:23:51.024757+0000","last_fullsized":"2024-03-30T13:23:51.024757+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:23:32.251853+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:23:32.251853+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:40.482169+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":902,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-03-30T13:23:52.172702+0000","last_change":"2024-03-30T13:23:38.517308+0000","last_active":"2024-03-30T13:23:52.172702+0000","last_peered":"2024-03-30T13:23:52.172702+0000","last_clean":"2024-03-30T13:23:52.172702+0000","last_became_active":"2024-03-30T13:23:38.516830+0000","last_became_peered":"2024-03-30T13:23:38.516830+0000","last_unstale":"2024-03-30T13:23:52.172702+0000","last_undegraded":"2024-03-30T13:23:52.172702+0000","last_fullsized":"2024-03-30T13:23:52.172702+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:23:25.785886+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:23:25.785886+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:26.017159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":264874756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387737,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58660,"kb_used_data":21464,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94313180,"statfs":{"total":96636764160,"available":96576696320,"internally_reserved":0,"allocated":21979136,"data_stored":18421507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157553,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63880,"kb_used_data":26232,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94307960,"statfs":{"total":96636764160,"available":96571351040,"internally_reserved":0,"allocated":26861568,"data_stored":23290337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":677,"seq":2907692859438,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59748,"kb_used_data":17936,"kb_used_omap":13,"kb_used_meta":41778,"kb_avail":94312092,"statfs":{"total":96636764160,"available":96575582208,"internally_reserved":0,"allocated":18366464,"data_stored":14795233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14287,"internal_metadata":42780721},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509783,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48040,"kb_used_data":12692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323800,"statfs":{"total":96636764160,"available":96587571200,"internally_reserved":0,"allocated":12996608,"data_stored":9432178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":769,"seq":3302829850629,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49624,"kb_used_data":11980,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94322216,"statfs":{"total":96636764160,"available":96585949184,"internally_reserved":0,"allocated":12267520,"data_stored":8696289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10995,"internal_metadata":38524173},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509784,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55204,"kb_used_data":19856,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94316636,"statfs":{"total":96636764160,"available":96580235264,"internally_reserved":0,"allocated":20332544,"data_stored":16774915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742457,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57020,"kb_used_data":16424,"kb_used_omap":15,"kb_used_meta":40560,"kb_avail":94314820,"statfs":{"total":96636764160,"available":96578375680,"internally_reserved":0,"allocated":16818176,"data_stored":13246945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":41533891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570127,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55636,"kb_used_data":18880,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94316204,"statfs":{"total":96636764160,"available":96579792896,"internally_reserved":0,"allocated":19333120,"data_stored":15761889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:23:55.550 INFO:tasks.thrashosds.thrasher:Choosing PG 1.5 with acting set [4, 2, 6] 2024-03-30T13:23:55.550 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [0, 2, 5, 7, 4, 6, 3, 1] 2024-03-30T13:23:55.550 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-30T13:23:55.550 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:23:55.615 INFO:tasks.ceph.osd.4:Stopped 2024-03-30T13:23:55.615 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 2, 5, 7, 4, 6, 3, 1] 2024-03-30T13:23:55.616 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 4 2024-03-30T13:23:55.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:55.738+0000 7fd7460eb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:55.758+0000 7fd7460eb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:55.934 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:23:55.934+0000 7fbb38f08640 -1 mon.a@0(leader).osd e786 definitely_dead 0 2024-03-30T13:23:56.654 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.4. 2024-03-30T13:23:56.681 INFO:tasks.thrashosds.thrasher:dead_osds=[4], live_osds=[0, 2, 5, 7, 6, 3, 1] 2024-03-30T13:23:56.682 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:23:56.682 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:23:56.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:56.802+0000 7fd5731da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:56.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:23:56.814+0000 7fd5731da640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:23:56.989 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:23:56.989 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:23:57.008 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1632,"stamp":"2024-03-30T13:23:56.649613+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":264874756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":388268,"kb_used_data":127744,"kb_used_omap":56,"kb_used_meta":260359,"kb_avail":660214612,"statfs":{"total":676457349120,"available":676059762688,"internally_reserved":0,"allocated":130809856,"data_stored":105723002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58158,"internal_metadata":266607826},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.375714"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1728,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-03-30T13:23:53.926137+0000","last_change":"2024-03-30T13:22:56.778810+0000","last_active":"2024-03-30T13:23:53.926137+0000","last_peered":"2024-03-30T13:23:53.926137+0000","last_clean":"2024-03-30T13:23:53.926137+0000","last_became_active":"2024-03-30T13:21:56.537232+0000","last_became_peered":"2024-03-30T13:21:56.537232+0000","last_unstale":"2024-03-30T13:23:53.926137+0000","last_undegraded":"2024-03-30T13:23:53.926137+0000","last_fullsized":"2024-03-30T13:23:53.926137+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:22:56.778530+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:22:56.778530+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:21.357262+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4588245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1679,"reported_epoch":783,"state":"stale+active+clean","last_fresh":"2024-03-30T13:23:53.378519+0000","last_change":"2024-03-30T13:23:30.947379+0000","last_active":"2024-03-30T13:23:53.378519+0000","last_peered":"2024-03-30T13:23:53.378519+0000","last_clean":"2024-03-30T13:23:53.378519+0000","last_became_active":"2024-03-30T13:22:47.491927+0000","last_became_peered":"2024-03-30T13:22:47.491927+0000","last_unstale":"2024-03-30T13:23:55.647125+0000","last_undegraded":"2024-03-30T13:23:53.378519+0000","last_fullsized":"2024-03-30T13:23:53.378519+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:30.947058+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:23:30.947058+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:31.095025+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":441,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:50.949900+0000","last_change":"2024-03-30T13:23:38.489520+0000","last_active":"2024-03-30T13:23:50.949900+0000","last_peered":"2024-03-30T13:23:50.949900+0000","last_clean":"2024-03-30T13:23:50.949900+0000","last_became_active":"2024-03-30T13:23:38.488514+0000","last_became_peered":"2024-03-30T13:23:38.488514+0000","last_unstale":"2024-03-30T13:23:50.949900+0000","last_undegraded":"2024-03-30T13:23:50.949900+0000","last_fullsized":"2024-03-30T13:23:50.949900+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:23:20.324157+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:23:20.324157+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:48.033710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":434,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:50.949725+0000","last_change":"2024-03-30T13:23:36.180822+0000","last_active":"2024-03-30T13:23:50.949725+0000","last_peered":"2024-03-30T13:23:50.949725+0000","last_clean":"2024-03-30T13:23:50.949725+0000","last_became_active":"2024-03-30T13:21:56.541153+0000","last_became_peered":"2024-03-30T13:21:56.541153+0000","last_unstale":"2024-03-30T13:23:50.949725+0000","last_undegraded":"2024-03-30T13:23:50.949725+0000","last_fullsized":"2024-03-30T13:23:50.949725+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:23:36.180557+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:52.909200+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":607,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:51.024749+0000","last_change":"2024-03-30T13:23:23.425617+0000","last_active":"2024-03-30T13:23:51.024749+0000","last_peered":"2024-03-30T13:23:51.024749+0000","last_clean":"2024-03-30T13:23:51.024749+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:23:51.024749+0000","last_undegraded":"2024-03-30T13:23:51.024749+0000","last_fullsized":"2024-03-30T13:23:51.024749+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:23.425236+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:23:23.425236+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:24.000232+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1684,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-03-30T13:23:55.651310+0000","last_change":"2024-03-30T13:23:40.597999+0000","last_active":"2024-03-30T13:23:55.651310+0000","last_peered":"2024-03-30T13:23:55.651310+0000","last_clean":"2024-03-30T13:23:55.651310+0000","last_became_active":"2024-03-30T13:22:47.491313+0000","last_became_peered":"2024-03-30T13:22:47.491313+0000","last_unstale":"2024-03-30T13:23:55.651310+0000","last_undegraded":"2024-03-30T13:23:55.651310+0000","last_fullsized":"2024-03-30T13:23:55.651310+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:23:40.597839+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:23:40.597839+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:00.377458+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":469,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:50.949760+0000","last_change":"2024-03-30T13:23:22.286357+0000","last_active":"2024-03-30T13:23:50.949760+0000","last_peered":"2024-03-30T13:23:50.949760+0000","last_clean":"2024-03-30T13:23:50.949760+0000","last_became_active":"2024-03-30T13:20:14.421760+0000","last_became_peered":"2024-03-30T13:20:14.421760+0000","last_unstale":"2024-03-30T13:23:50.949760+0000","last_undegraded":"2024-03-30T13:23:50.949760+0000","last_fullsized":"2024-03-30T13:23:50.949760+0000","mapping_epoch":677,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":678,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:23:22.286071+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:23:22.286071+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:25.807487+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":29277398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1039,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-03-30T13:23:52.172637+0000","last_change":"2024-03-30T13:23:38.515178+0000","last_active":"2024-03-30T13:23:52.172637+0000","last_peered":"2024-03-30T13:23:52.172637+0000","last_clean":"2024-03-30T13:23:52.172637+0000","last_became_active":"2024-03-30T13:23:38.514736+0000","last_became_peered":"2024-03-30T13:23:38.514736+0000","last_unstale":"2024-03-30T13:23:52.172637+0000","last_undegraded":"2024-03-30T13:23:52.172637+0000","last_fullsized":"2024-03-30T13:23:52.172637+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:23:02.469679+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:23:02.469679+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:23.725884+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1787,"reported_epoch":786,"state":"active+clean","last_fresh":"2024-03-30T13:23:55.651140+0000","last_change":"2024-03-30T13:23:27.681044+0000","last_active":"2024-03-30T13:23:55.651140+0000","last_peered":"2024-03-30T13:23:55.651140+0000","last_clean":"2024-03-30T13:23:55.651140+0000","last_became_active":"2024-03-30T13:22:47.491356+0000","last_became_peered":"2024-03-30T13:22:47.491356+0000","last_unstale":"2024-03-30T13:23:55.651140+0000","last_undegraded":"2024-03-30T13:23:55.651140+0000","last_fullsized":"2024-03-30T13:23:55.651140+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:23:27.680784+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:45.246430+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1501,"reported_epoch":783,"state":"stale+active+clean","last_fresh":"2024-03-30T13:23:53.378141+0000","last_change":"2024-03-30T13:23:10.054085+0000","last_active":"2024-03-30T13:23:53.378141+0000","last_peered":"2024-03-30T13:23:53.378141+0000","last_clean":"2024-03-30T13:23:53.378141+0000","last_became_active":"2024-03-30T13:21:56.531330+0000","last_became_peered":"2024-03-30T13:21:56.531330+0000","last_unstale":"2024-03-30T13:23:55.647129+0000","last_undegraded":"2024-03-30T13:23:53.378141+0000","last_fullsized":"2024-03-30T13:23:53.378141+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:10.053837+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:23:10.053837+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:22.003357+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4672280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1755,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-03-30T13:23:53.925656+0000","last_change":"2024-03-30T13:23:19.612300+0000","last_active":"2024-03-30T13:23:53.925656+0000","last_peered":"2024-03-30T13:23:53.925656+0000","last_clean":"2024-03-30T13:23:53.925656+0000","last_became_active":"2024-03-30T13:21:56.537406+0000","last_became_peered":"2024-03-30T13:21:56.537406+0000","last_unstale":"2024-03-30T13:23:53.925656+0000","last_undegraded":"2024-03-30T13:23:53.925656+0000","last_fullsized":"2024-03-30T13:23:53.925656+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:23:19.611935+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:23:19.611935+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:48.677040+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1842,"reported_epoch":785,"state":"active+clean","last_fresh":"2024-03-30T13:23:53.925951+0000","last_change":"2024-03-30T13:23:22.573707+0000","last_active":"2024-03-30T13:23:53.925951+0000","last_peered":"2024-03-30T13:23:53.925951+0000","last_clean":"2024-03-30T13:23:53.925951+0000","last_became_active":"2024-03-30T13:22:47.488417+0000","last_became_peered":"2024-03-30T13:22:47.488417+0000","last_unstale":"2024-03-30T13:23:53.925951+0000","last_undegraded":"2024-03-30T13:23:53.925951+0000","last_fullsized":"2024-03-30T13:23:53.925951+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:23:22.573400+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:43.538364+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":662,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:51.024723+0000","last_change":"2024-03-30T13:22:53.439552+0000","last_active":"2024-03-30T13:23:51.024723+0000","last_peered":"2024-03-30T13:23:51.024723+0000","last_clean":"2024-03-30T13:23:51.024723+0000","last_became_active":"2024-03-30T13:22:47.493033+0000","last_became_peered":"2024-03-30T13:22:47.493033+0000","last_unstale":"2024-03-30T13:23:51.024723+0000","last_undegraded":"2024-03-30T13:23:51.024723+0000","last_fullsized":"2024-03-30T13:23:51.024723+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:22:53.439303+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:22:53.439303+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:10.009027+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18319366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":438,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:51.071558+0000","last_change":"2024-03-30T13:23:51.071627+0000","last_active":"2024-03-30T13:23:51.071558+0000","last_peered":"2024-03-30T13:23:51.071558+0000","last_clean":"2024-03-30T13:23:51.071558+0000","last_became_active":"2024-03-30T13:22:47.489008+0000","last_became_peered":"2024-03-30T13:22:47.489008+0000","last_unstale":"2024-03-30T13:23:51.071558+0000","last_undegraded":"2024-03-30T13:23:51.071558+0000","last_fullsized":"2024-03-30T13:23:51.071558+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:51.071390+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:23:51.071390+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:10.481199+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4974806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":647,"reported_epoch":782,"state":"active+clean","last_fresh":"2024-03-30T13:23:51.024757+0000","last_change":"2024-03-30T13:23:32.252109+0000","last_active":"2024-03-30T13:23:51.024757+0000","last_peered":"2024-03-30T13:23:51.024757+0000","last_clean":"2024-03-30T13:23:51.024757+0000","last_became_active":"2024-03-30T13:21:56.534725+0000","last_became_peered":"2024-03-30T13:21:56.534725+0000","last_unstale":"2024-03-30T13:23:51.024757+0000","last_undegraded":"2024-03-30T13:23:51.024757+0000","last_fullsized":"2024-03-30T13:23:51.024757+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:23:32.251853+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:23:32.251853+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:40.482169+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":902,"reported_epoch":783,"state":"active+clean","last_fresh":"2024-03-30T13:23:52.172702+0000","last_change":"2024-03-30T13:23:38.517308+0000","last_active":"2024-03-30T13:23:52.172702+0000","last_peered":"2024-03-30T13:23:52.172702+0000","last_clean":"2024-03-30T13:23:52.172702+0000","last_became_active":"2024-03-30T13:23:38.516830+0000","last_became_peered":"2024-03-30T13:23:38.516830+0000","last_unstale":"2024-03-30T13:23:52.172702+0000","last_undegraded":"2024-03-30T13:23:52.172702+0000","last_fullsized":"2024-03-30T13:23:52.172702+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:23:25.785886+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:23:25.785886+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:26.017159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":264874756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387738,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58728,"kb_used_data":21524,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94313112,"statfs":{"total":96636764160,"available":96576626688,"internally_reserved":0,"allocated":22040576,"data_stored":18448942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157553,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63880,"kb_used_data":26232,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94307960,"statfs":{"total":96636764160,"available":96571351040,"internally_reserved":0,"allocated":26861568,"data_stored":23290337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509784,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48080,"kb_used_data":12740,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323760,"statfs":{"total":96636764160,"available":96587530240,"internally_reserved":0,"allocated":13045760,"data_stored":9454126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":769,"seq":3302829850629,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49624,"kb_used_data":11980,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94322216,"statfs":{"total":96636764160,"available":96585949184,"internally_reserved":0,"allocated":12267520,"data_stored":8696289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10995,"internal_metadata":38524173},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509785,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55260,"kb_used_data":19916,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94316580,"statfs":{"total":96636764160,"available":96580177920,"internally_reserved":0,"allocated":20393984,"data_stored":16802350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742457,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57020,"kb_used_data":16424,"kb_used_omap":15,"kb_used_meta":40560,"kb_avail":94314820,"statfs":{"total":96636764160,"available":96578375680,"internally_reserved":0,"allocated":16818176,"data_stored":13246945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":41533891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570128,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55676,"kb_used_data":18928,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94316164,"statfs":{"total":96636764160,"available":96579751936,"internally_reserved":0,"allocated":19382272,"data_stored":15784013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:23:57.009 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:24:22.009 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:24:22.010 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:24:22.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:22.142+0000 7f0ab2395640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:22.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:22.158+0000 7f0ab2395640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:22.329 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:24:22.329 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:24:22.347 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1650,"stamp":"2024-03-30T13:24:22.285792+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":297009311,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":401140,"kb_used_data":140640,"kb_used_omap":56,"kb_used_meta":260359,"kb_avail":660201740,"statfs":{"total":676457349120,"available":676046581760,"internally_reserved":0,"allocated":144015360,"data_stored":118543767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58158,"internal_metadata":266607826},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002876"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1762,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-03-30T13:24:15.606371+0000","last_change":"2024-03-30T13:24:01.591716+0000","last_active":"2024-03-30T13:24:15.606371+0000","last_peered":"2024-03-30T13:24:15.606371+0000","last_clean":"2024-03-30T13:24:15.606371+0000","last_became_active":"2024-03-30T13:23:57.672452+0000","last_became_peered":"2024-03-30T13:23:57.672452+0000","last_unstale":"2024-03-30T13:24:15.606371+0000","last_undegraded":"2024-03-30T13:24:15.606371+0000","last_fullsized":"2024-03-30T13:24:15.606371+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:24:01.591463+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:24:01.591463+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:07.705451+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":23,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-03-30T13:24:15.388873+0000","last_change":"2024-03-30T13:23:57.891744+0000","last_active":"2024-03-30T13:24:15.388873+0000","last_peered":"2024-03-30T13:24:15.388873+0000","last_clean":"2024-03-30T13:24:15.388873+0000","last_became_active":"2024-03-30T13:23:57.667838+0000","last_became_peered":"2024-03-30T13:23:57.667838+0000","last_unstale":"2024-03-30T13:24:15.388873+0000","last_undegraded":"2024-03-30T13:24:15.388873+0000","last_fullsized":"2024-03-30T13:24:15.388873+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:30.947058+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:23:30.947058+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:36.800055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":474,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-03-30T13:24:15.882883+0000","last_change":"2024-03-30T13:23:59.351454+0000","last_active":"2024-03-30T13:24:15.882883+0000","last_peered":"2024-03-30T13:24:15.882883+0000","last_clean":"2024-03-30T13:24:15.882883+0000","last_became_active":"2024-03-30T13:23:57.669676+0000","last_became_peered":"2024-03-30T13:23:57.669676+0000","last_unstale":"2024-03-30T13:24:15.882883+0000","last_undegraded":"2024-03-30T13:24:15.882883+0000","last_fullsized":"2024-03-30T13:24:15.882883+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:23:20.324157+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:23:20.324157+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:26.931161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":39075780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":454,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-03-30T13:24:15.882829+0000","last_change":"2024-03-30T13:23:36.180822+0000","last_active":"2024-03-30T13:24:15.882829+0000","last_peered":"2024-03-30T13:24:15.882829+0000","last_clean":"2024-03-30T13:24:15.882829+0000","last_became_active":"2024-03-30T13:21:56.541153+0000","last_became_peered":"2024-03-30T13:21:56.541153+0000","last_unstale":"2024-03-30T13:24:15.882829+0000","last_undegraded":"2024-03-30T13:24:15.882829+0000","last_fullsized":"2024-03-30T13:24:15.882829+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:23:36.180557+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:52.909200+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":626,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-03-30T13:24:15.387378+0000","last_change":"2024-03-30T13:23:23.425617+0000","last_active":"2024-03-30T13:24:15.387378+0000","last_peered":"2024-03-30T13:24:15.387378+0000","last_clean":"2024-03-30T13:24:15.387378+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:24:15.387378+0000","last_undegraded":"2024-03-30T13:24:15.387378+0000","last_fullsized":"2024-03-30T13:24:15.387378+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:23.425236+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:23:23.425236+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:24.000232+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1696,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-03-30T13:24:15.389756+0000","last_change":"2024-03-30T13:23:40.597999+0000","last_active":"2024-03-30T13:24:15.389756+0000","last_peered":"2024-03-30T13:24:15.389756+0000","last_clean":"2024-03-30T13:24:15.389756+0000","last_became_active":"2024-03-30T13:22:47.491313+0000","last_became_peered":"2024-03-30T13:22:47.491313+0000","last_unstale":"2024-03-30T13:24:15.389756+0000","last_undegraded":"2024-03-30T13:24:15.389756+0000","last_fullsized":"2024-03-30T13:24:15.389756+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:23:40.597839+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:23:40.597839+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:00.377458+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":508,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-03-30T13:24:15.882820+0000","last_change":"2024-03-30T13:23:58.741239+0000","last_active":"2024-03-30T13:24:15.882820+0000","last_peered":"2024-03-30T13:24:15.882820+0000","last_clean":"2024-03-30T13:24:15.882820+0000","last_became_active":"2024-03-30T13:23:57.670170+0000","last_became_peered":"2024-03-30T13:23:57.670170+0000","last_unstale":"2024-03-30T13:24:15.882820+0000","last_undegraded":"2024-03-30T13:24:15.882820+0000","last_fullsized":"2024-03-30T13:24:15.882820+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":789,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:23:22.286071+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:23:22.286071+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:47.339102+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":43916097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1057,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-03-30T13:24:15.611951+0000","last_change":"2024-03-30T13:23:38.515178+0000","last_active":"2024-03-30T13:24:15.611951+0000","last_peered":"2024-03-30T13:24:15.611951+0000","last_clean":"2024-03-30T13:24:15.611951+0000","last_became_active":"2024-03-30T13:23:38.514736+0000","last_became_peered":"2024-03-30T13:23:38.514736+0000","last_unstale":"2024-03-30T13:24:15.611951+0000","last_undegraded":"2024-03-30T13:24:15.611951+0000","last_fullsized":"2024-03-30T13:24:15.611951+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:23:02.469679+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:23:02.469679+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:23.725884+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1799,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-03-30T13:24:15.389907+0000","last_change":"2024-03-30T13:23:27.681044+0000","last_active":"2024-03-30T13:24:15.389907+0000","last_peered":"2024-03-30T13:24:15.389907+0000","last_clean":"2024-03-30T13:24:15.389907+0000","last_became_active":"2024-03-30T13:22:47.491356+0000","last_became_peered":"2024-03-30T13:22:47.491356+0000","last_unstale":"2024-03-30T13:24:15.389907+0000","last_undegraded":"2024-03-30T13:24:15.389907+0000","last_fullsized":"2024-03-30T13:24:15.389907+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:23:27.680784+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:45.246430+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":23,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-03-30T13:24:15.612567+0000","last_change":"2024-03-30T13:23:57.702479+0000","last_active":"2024-03-30T13:24:15.612567+0000","last_peered":"2024-03-30T13:24:15.612567+0000","last_clean":"2024-03-30T13:24:15.612567+0000","last_became_active":"2024-03-30T13:23:57.666371+0000","last_became_peered":"2024-03-30T13:23:57.666371+0000","last_unstale":"2024-03-30T13:24:15.612567+0000","last_undegraded":"2024-03-30T13:24:15.612567+0000","last_fullsized":"2024-03-30T13:24:15.612567+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:10.053837+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:23:10.053837+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:22.154159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1787,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-03-30T13:24:15.606216+0000","last_change":"2024-03-30T13:23:58.015288+0000","last_active":"2024-03-30T13:24:15.606216+0000","last_peered":"2024-03-30T13:24:15.606216+0000","last_clean":"2024-03-30T13:24:15.606216+0000","last_became_active":"2024-03-30T13:23:57.665346+0000","last_became_peered":"2024-03-30T13:23:57.665346+0000","last_unstale":"2024-03-30T13:24:15.606216+0000","last_undegraded":"2024-03-30T13:24:15.606216+0000","last_fullsized":"2024-03-30T13:24:15.606216+0000","mapping_epoch":787,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":788,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:23:19.611935+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:23:19.611935+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:48.143125+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1857,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-03-30T13:24:15.606014+0000","last_change":"2024-03-30T13:23:22.573707+0000","last_active":"2024-03-30T13:24:15.606014+0000","last_peered":"2024-03-30T13:24:15.606014+0000","last_clean":"2024-03-30T13:24:15.606014+0000","last_became_active":"2024-03-30T13:22:47.488417+0000","last_became_peered":"2024-03-30T13:22:47.488417+0000","last_unstale":"2024-03-30T13:24:15.606014+0000","last_undegraded":"2024-03-30T13:24:15.606014+0000","last_fullsized":"2024-03-30T13:24:15.606014+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:23:22.573400+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:43.538364+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":689,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-03-30T13:24:15.387381+0000","last_change":"2024-03-30T13:24:10.389590+0000","last_active":"2024-03-30T13:24:15.387381+0000","last_peered":"2024-03-30T13:24:15.387381+0000","last_clean":"2024-03-30T13:24:15.387381+0000","last_became_active":"2024-03-30T13:22:47.493033+0000","last_became_peered":"2024-03-30T13:22:47.493033+0000","last_unstale":"2024-03-30T13:24:15.387381+0000","last_undegraded":"2024-03-30T13:24:15.387381+0000","last_fullsized":"2024-03-30T13:24:15.387381+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:24:10.389317+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:24:10.389317+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:17.072573+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18319366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":458,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-03-30T13:24:15.882683+0000","last_change":"2024-03-30T13:23:51.071627+0000","last_active":"2024-03-30T13:24:15.882683+0000","last_peered":"2024-03-30T13:24:15.882683+0000","last_clean":"2024-03-30T13:24:15.882683+0000","last_became_active":"2024-03-30T13:22:47.489008+0000","last_became_peered":"2024-03-30T13:22:47.489008+0000","last_unstale":"2024-03-30T13:24:15.882683+0000","last_undegraded":"2024-03-30T13:24:15.882683+0000","last_fullsized":"2024-03-30T13:24:15.882683+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:51.071390+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:23:51.071390+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:10.481199+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4974806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":666,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-03-30T13:24:15.387400+0000","last_change":"2024-03-30T13:23:32.252109+0000","last_active":"2024-03-30T13:24:15.387400+0000","last_peered":"2024-03-30T13:24:15.387400+0000","last_clean":"2024-03-30T13:24:15.387400+0000","last_became_active":"2024-03-30T13:21:56.534725+0000","last_became_peered":"2024-03-30T13:21:56.534725+0000","last_unstale":"2024-03-30T13:24:15.387400+0000","last_undegraded":"2024-03-30T13:24:15.387400+0000","last_fullsized":"2024-03-30T13:24:15.387400+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:23:32.251853+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:10:48.605866+0000","last_clean_scrub_stamp":"2024-03-30T13:23:32.251853+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:40.482169+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":920,"reported_epoch":792,"state":"active+clean","last_fresh":"2024-03-30T13:24:15.612029+0000","last_change":"2024-03-30T13:23:38.517308+0000","last_active":"2024-03-30T13:24:15.612029+0000","last_peered":"2024-03-30T13:24:15.612029+0000","last_clean":"2024-03-30T13:24:15.612029+0000","last_became_active":"2024-03-30T13:23:38.516830+0000","last_became_peered":"2024-03-30T13:23:38.516830+0000","last_unstale":"2024-03-30T13:24:15.612029+0000","last_undegraded":"2024-03-30T13:24:15.612029+0000","last_fullsized":"2024-03-30T13:24:15.612029+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:23:25.785886+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:23:25.785886+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:26.017159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":297009311,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":109256704,"data_stored":109256704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387743,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58804,"kb_used_data":21608,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94313036,"statfs":{"total":96636764160,"available":96576548864,"internally_reserved":0,"allocated":22126592,"data_stored":18487793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157558,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63996,"kb_used_data":26352,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94307844,"statfs":{"total":96636764160,"available":96571232256,"internally_reserved":0,"allocated":26984448,"data_stored":23345649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509789,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54920,"kb_used_data":19580,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94316920,"statfs":{"total":96636764160,"available":96580526080,"internally_reserved":0,"allocated":20049920,"data_stored":16411121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":769,"seq":3302829850634,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50836,"kb_used_data":13192,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94321004,"statfs":{"total":96636764160,"available":96584708096,"internally_reserved":0,"allocated":13508608,"data_stored":9869809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10995,"internal_metadata":38524173},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509790,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55844,"kb_used_data":20504,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315996,"statfs":{"total":96636764160,"available":96579579904,"internally_reserved":0,"allocated":20996096,"data_stored":17357297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742462,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57132,"kb_used_data":16544,"kb_used_omap":15,"kb_used_meta":40560,"kb_avail":94314708,"statfs":{"total":96636764160,"available":96578260992,"internally_reserved":0,"allocated":16941056,"data_stored":13302257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":41533891},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570133,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59608,"kb_used_data":22860,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94312232,"statfs":{"total":96636764160,"available":96575725568,"internally_reserved":0,"allocated":23408640,"data_stored":19769841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14340096,"data_stored":14340096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":6852608,"data_stored":6852608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":13393920,"data_stored":13393920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16752640,"data_stored":16752640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:24:22.348 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:24:22.348 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:24:22.349 INFO:tasks.thrashosds.thrasher:Reviving osd 4 2024-03-30T13:24:22.349 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-30T13:24:22.349 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-30T13:24:22.351 INFO:tasks.ceph.osd.4:Started 2024-03-30T13:24:22.352 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-30T13:24:22.421 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:24:22.418+0000 7ff7bf6b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:22.431 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:24:22.430+0000 7ff7bf6b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:22.431 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:24:22.430+0000 7ff7bf6b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:22.476 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T13:24:22.476 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T13:24:22.476 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T13:24:22.476 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T13:24:22.476 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T13:24:22.476 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T13:24:22.476 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-03-30T13:24:22.477 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-03-30T13:24:22.477 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-03-30T13:24:22.477 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-03-30T13:24:22.477 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-03-30T13:24:22.477 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T13:24:22.478 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:24:22.479 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-03-30T13:24:22.962 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:24:22.962+0000 7ff7bf6b57c0 -1 Falling back to public interface 2024-03-30T13:24:26.892 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:24:26.890+0000 7ff7bf6b57c0 -1 osd.4 785 log_to_monitors true 2024-03-30T13:24:27.446 INFO:tasks.ceph.osd.4.smithi082.stderr:2024-03-30T13:24:27.446+0000 7ff7b3482640 -1 osd.4 785 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:24:27.480 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-03-30T13:24:27.610 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T13:24:27.611 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T13:24:27.611 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T13:24:27.611 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T13:24:27.624 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:24:27.624 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-03-30T13:24:27.625 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 4 2024-03-30T13:24:27.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:27.753+0000 7fe515d51640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:27.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:27.769+0000 7fe515d51640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:27.948 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:24:27.949+0000 7fbb38f08640 -1 mon.a@0(leader).osd e793 definitely_dead 0 2024-03-30T13:24:28.423 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.4. 2024-03-30T13:24:28.457 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-03-30T13:24:28.457 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:24:28.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:28.569+0000 7f4fa1cd5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:28.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:28.585+0000 7f4fa1cd5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:29.432 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:24:29.465 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:24:29.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:29.585+0000 7f9f635aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:29.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:29.601+0000 7f9f635aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:30.453 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:24:30.478 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:24:30.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:30.597+0000 7f68397f4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:30.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:30.613+0000 7f68397f4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:31.455 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:24:31.486 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:24:31.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:31.609+0000 7f893679a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:31.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:31.625+0000 7f893679a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:32.479 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:24:32.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:24:32.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:32.625+0000 7f0c70557640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:32.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:32.641+0000 7f0c70557640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:33.485 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:24:33.511 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:24:33.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:33.633+0000 7f4bfe96d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:33.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:33.649+0000 7f4bfe96d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:34.501 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:24:34.528 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:24:34.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:34.649+0000 7fdf532d2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:34.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:34.665+0000 7fdf532d2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:35.516 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:24:35.548 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:24:35.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:35.669+0000 7fd711ee3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:35.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:35.685+0000 7fd711ee3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:36.551 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:24:36.577 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:24:36.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:36.701+0000 7fee4e68f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:36.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:36.717+0000 7fee4e68f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:37.555 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:24:37.584 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:24:37.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:37.709+0000 7f6d6e1e8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:37.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:37.725+0000 7f6d6e1e8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:38.558 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:24:38.586 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:24:38.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:38.705+0000 7f7b2ef29640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:38.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:38.721+0000 7f7b2ef29640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:39.580 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:24:39.608 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:24:39.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:39.729+0000 7f964d120640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:39.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:39.745+0000 7f964d120640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:40.590 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:24:40.613 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:24:40.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:40.737+0000 7f98f2208640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:40.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:40.753+0000 7f98f2208640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:41.607 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:24:41.630 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:24:41.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:41.753+0000 7f6ed5098640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:41.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:41.769+0000 7f6ed5098640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:42.616 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:24:42.641 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:24:42.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:42.761+0000 7f4def679640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:42.781+0000 7f4def679640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:43.622 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:24:43.648 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:24:43.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:43.761+0000 7f3fd662b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:43.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:43.777+0000 7f3fd662b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:44.635 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:24:44.662 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:24:44.663 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:24:44.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:44.785+0000 7fa77833e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:44.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:44.801+0000 7fa77833e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:44.971 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:24:44.971 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:24:44.989 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1679,"stamp":"2024-03-30T13:24:44.631411+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":297009311,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":448852,"kb_used_data":147792,"kb_used_omap":72,"kb_used_meta":300855,"kb_avail":754525868,"statfs":{"total":773094113280,"available":772634488832,"internally_reserved":0,"allocated":151339008,"data_stored":121500179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74099,"internal_metadata":308076173},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.043042"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1794,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-03-30T13:24:40.104810+0000","last_change":"2024-03-30T13:24:29.448256+0000","last_active":"2024-03-30T13:24:40.104810+0000","last_peered":"2024-03-30T13:24:40.104810+0000","last_clean":"2024-03-30T13:24:40.104810+0000","last_became_active":"2024-03-30T13:24:29.446987+0000","last_became_peered":"2024-03-30T13:24:29.446987+0000","last_unstale":"2024-03-30T13:24:40.104810+0000","last_undegraded":"2024-03-30T13:24:40.104810+0000","last_fullsized":"2024-03-30T13:24:40.104810+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:24:01.591463+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:24:01.591463+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:10.966108+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1683,"reported_epoch":808,"state":"active+clean","last_fresh":"2024-03-30T13:24:42.615283+0000","last_change":"2024-03-30T13:24:29.440117+0000","last_active":"2024-03-30T13:24:42.615283+0000","last_peered":"2024-03-30T13:24:42.615283+0000","last_clean":"2024-03-30T13:24:42.615283+0000","last_became_active":"2024-03-30T13:24:29.438667+0000","last_became_peered":"2024-03-30T13:24:29.438667+0000","last_unstale":"2024-03-30T13:24:42.615283+0000","last_undegraded":"2024-03-30T13:24:42.615283+0000","last_fullsized":"2024-03-30T13:24:42.615283+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:30.947058+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:23:30.947058+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:58.949105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":515,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-03-30T13:24:41.214789+0000","last_change":"2024-03-30T13:24:29.442221+0000","last_active":"2024-03-30T13:24:41.214789+0000","last_peered":"2024-03-30T13:24:41.214789+0000","last_clean":"2024-03-30T13:24:41.214789+0000","last_became_active":"2024-03-30T13:24:29.440460+0000","last_became_peered":"2024-03-30T13:24:29.440460+0000","last_unstale":"2024-03-30T13:24:41.214789+0000","last_undegraded":"2024-03-30T13:24:41.214789+0000","last_fullsized":"2024-03-30T13:24:41.214789+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:24:27.040925+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:24:27.040925+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:51.105436+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":39075780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":481,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-03-30T13:24:41.214693+0000","last_change":"2024-03-30T13:23:36.180822+0000","last_active":"2024-03-30T13:24:41.214693+0000","last_peered":"2024-03-30T13:24:41.214693+0000","last_clean":"2024-03-30T13:24:41.214693+0000","last_became_active":"2024-03-30T13:21:56.541153+0000","last_became_peered":"2024-03-30T13:21:56.541153+0000","last_unstale":"2024-03-30T13:24:41.214693+0000","last_undegraded":"2024-03-30T13:24:41.214693+0000","last_fullsized":"2024-03-30T13:24:41.214693+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:23:36.180557+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:52.909200+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":662,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-03-30T13:24:40.619528+0000","last_change":"2024-03-30T13:24:24.431877+0000","last_active":"2024-03-30T13:24:40.619528+0000","last_peered":"2024-03-30T13:24:40.619528+0000","last_clean":"2024-03-30T13:24:40.619528+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:24:40.619528+0000","last_undegraded":"2024-03-30T13:24:40.619528+0000","last_fullsized":"2024-03-30T13:24:40.619528+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:24:24.431636+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:24:24.431636+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:46.235372+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1724,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-03-30T13:24:40.592747+0000","last_change":"2024-03-30T13:23:40.597999+0000","last_active":"2024-03-30T13:24:40.592747+0000","last_peered":"2024-03-30T13:24:40.592747+0000","last_clean":"2024-03-30T13:24:40.592747+0000","last_became_active":"2024-03-30T13:22:47.491313+0000","last_became_peered":"2024-03-30T13:22:47.491313+0000","last_unstale":"2024-03-30T13:24:40.592747+0000","last_undegraded":"2024-03-30T13:24:40.592747+0000","last_fullsized":"2024-03-30T13:24:40.592747+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:23:40.597839+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:23:40.597839+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:00.377458+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":549,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-03-30T13:24:41.214736+0000","last_change":"2024-03-30T13:24:32.092922+0000","last_active":"2024-03-30T13:24:41.214736+0000","last_peered":"2024-03-30T13:24:41.214736+0000","last_clean":"2024-03-30T13:24:41.214736+0000","last_became_active":"2024-03-30T13:24:29.440149+0000","last_became_peered":"2024-03-30T13:24:29.440149+0000","last_unstale":"2024-03-30T13:24:41.214736+0000","last_undegraded":"2024-03-30T13:24:41.214736+0000","last_fullsized":"2024-03-30T13:24:41.214736+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:24:32.092620+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:24:32.092620+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:42.584351+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":43916097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1095,"reported_epoch":807,"state":"active+clean","last_fresh":"2024-03-30T13:24:41.609310+0000","last_change":"2024-03-30T13:24:35.540725+0000","last_active":"2024-03-30T13:24:41.609310+0000","last_peered":"2024-03-30T13:24:41.609310+0000","last_clean":"2024-03-30T13:24:41.609310+0000","last_became_active":"2024-03-30T13:23:38.514736+0000","last_became_peered":"2024-03-30T13:23:38.514736+0000","last_unstale":"2024-03-30T13:24:41.609310+0000","last_undegraded":"2024-03-30T13:24:41.609310+0000","last_fullsized":"2024-03-30T13:24:41.609310+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:24:35.540394+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:24:35.540394+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:42.188887+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1827,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-03-30T13:24:40.592449+0000","last_change":"2024-03-30T13:23:27.681044+0000","last_active":"2024-03-30T13:24:40.592449+0000","last_peered":"2024-03-30T13:24:40.592449+0000","last_clean":"2024-03-30T13:24:40.592449+0000","last_became_active":"2024-03-30T13:22:47.491356+0000","last_became_peered":"2024-03-30T13:22:47.491356+0000","last_unstale":"2024-03-30T13:24:40.592449+0000","last_undegraded":"2024-03-30T13:24:40.592449+0000","last_fullsized":"2024-03-30T13:24:40.592449+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:23:27.680784+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:45.246430+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1513,"reported_epoch":808,"state":"active+clean","last_fresh":"2024-03-30T13:24:42.615145+0000","last_change":"2024-03-30T13:24:30.839444+0000","last_active":"2024-03-30T13:24:42.615145+0000","last_peered":"2024-03-30T13:24:42.615145+0000","last_clean":"2024-03-30T13:24:42.615145+0000","last_became_active":"2024-03-30T13:24:29.442802+0000","last_became_peered":"2024-03-30T13:24:29.442802+0000","last_unstale":"2024-03-30T13:24:42.615145+0000","last_undegraded":"2024-03-30T13:24:42.615145+0000","last_fullsized":"2024-03-30T13:24:42.615145+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:24:30.839164+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:24:30.839164+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:38.428401+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1819,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-03-30T13:24:40.104367+0000","last_change":"2024-03-30T13:24:29.448451+0000","last_active":"2024-03-30T13:24:40.104367+0000","last_peered":"2024-03-30T13:24:40.104367+0000","last_clean":"2024-03-30T13:24:40.104367+0000","last_became_active":"2024-03-30T13:24:29.447202+0000","last_became_peered":"2024-03-30T13:24:29.447202+0000","last_unstale":"2024-03-30T13:24:40.104367+0000","last_undegraded":"2024-03-30T13:24:40.104367+0000","last_fullsized":"2024-03-30T13:24:40.104367+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:23:19.611935+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:23:19.611935+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:43.473381+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1883,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-03-30T13:24:40.104374+0000","last_change":"2024-03-30T13:23:22.573707+0000","last_active":"2024-03-30T13:24:40.104374+0000","last_peered":"2024-03-30T13:24:40.104374+0000","last_clean":"2024-03-30T13:24:40.104374+0000","last_became_active":"2024-03-30T13:22:47.488417+0000","last_became_peered":"2024-03-30T13:22:47.488417+0000","last_unstale":"2024-03-30T13:24:40.104374+0000","last_undegraded":"2024-03-30T13:24:40.104374+0000","last_fullsized":"2024-03-30T13:24:40.104374+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:23:22.573400+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:43.538364+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":717,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-03-30T13:24:40.619478+0000","last_change":"2024-03-30T13:24:10.389590+0000","last_active":"2024-03-30T13:24:40.619478+0000","last_peered":"2024-03-30T13:24:40.619478+0000","last_clean":"2024-03-30T13:24:40.619478+0000","last_became_active":"2024-03-30T13:22:47.493033+0000","last_became_peered":"2024-03-30T13:22:47.493033+0000","last_unstale":"2024-03-30T13:24:40.619478+0000","last_undegraded":"2024-03-30T13:24:40.619478+0000","last_fullsized":"2024-03-30T13:24:40.619478+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:24:10.389317+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:24:10.389317+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:17.072573+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18319366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":485,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-03-30T13:24:41.214761+0000","last_change":"2024-03-30T13:23:51.071627+0000","last_active":"2024-03-30T13:24:41.214761+0000","last_peered":"2024-03-30T13:24:41.214761+0000","last_clean":"2024-03-30T13:24:41.214761+0000","last_became_active":"2024-03-30T13:22:47.489008+0000","last_became_peered":"2024-03-30T13:22:47.489008+0000","last_unstale":"2024-03-30T13:24:41.214761+0000","last_undegraded":"2024-03-30T13:24:41.214761+0000","last_fullsized":"2024-03-30T13:24:41.214761+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:51.071390+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:23:51.071390+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:10.481199+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4974806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":702,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-03-30T13:24:40.633815+0000","last_change":"2024-03-30T13:24:40.633893+0000","last_active":"2024-03-30T13:24:40.633815+0000","last_peered":"2024-03-30T13:24:40.633815+0000","last_clean":"2024-03-30T13:24:40.633815+0000","last_became_active":"2024-03-30T13:21:56.534725+0000","last_became_peered":"2024-03-30T13:21:56.534725+0000","last_unstale":"2024-03-30T13:24:40.633815+0000","last_undegraded":"2024-03-30T13:24:40.633815+0000","last_fullsized":"2024-03-30T13:24:40.633815+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_clean_scrub_stamp":"2024-03-30T13:24:40.633617+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:52.628929+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":958,"reported_epoch":807,"state":"active+clean","last_fresh":"2024-03-30T13:24:41.609413+0000","last_change":"2024-03-30T13:24:36.495425+0000","last_active":"2024-03-30T13:24:41.609413+0000","last_peered":"2024-03-30T13:24:41.609413+0000","last_clean":"2024-03-30T13:24:41.609413+0000","last_became_active":"2024-03-30T13:23:38.516830+0000","last_became_peered":"2024-03-30T13:23:38.516830+0000","last_unstale":"2024-03-30T13:24:41.609413+0000","last_undegraded":"2024-03-30T13:24:41.609413+0000","last_fullsized":"2024-03-30T13:24:41.609413+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:24:36.495187+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:24:36.495187+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:37.050877+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":297009311,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387747,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58956,"kb_used_data":21752,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94312884,"statfs":{"total":96636764160,"available":96576393216,"internally_reserved":0,"allocated":22274048,"data_stored":18555246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157563,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64176,"kb_used_data":26520,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94307664,"statfs":{"total":96636764160,"available":96571047936,"internally_reserved":0,"allocated":27156480,"data_stored":23424076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":794,"seq":3410204033029,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58772,"kb_used_data":18248,"kb_used_omap":15,"kb_used_meta":40496,"kb_avail":94313068,"statfs":{"total":96636764160,"available":96576581632,"internally_reserved":0,"allocated":18685952,"data_stored":14939946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15941,"internal_metadata":41468347},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509793,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48308,"kb_used_data":12968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323532,"statfs":{"total":96636764160,"available":96587296768,"internally_reserved":0,"allocated":13279232,"data_stored":9560430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":769,"seq":3302829850639,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49932,"kb_used_data":12280,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94321908,"statfs":{"total":96636764160,"available":96585633792,"internally_reserved":0,"allocated":12574720,"data_stored":8835515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10995,"internal_metadata":38524173},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509794,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55484,"kb_used_data":20144,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94316356,"statfs":{"total":96636764160,"available":96579948544,"internally_reserved":0,"allocated":20627456,"data_stored":16908654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742467,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57304,"kb_used_data":16712,"kb_used_omap":15,"kb_used_meta":40560,"kb_avail":94314536,"statfs":{"total":96636764160,"available":96578084864,"internally_reserved":0,"allocated":17113088,"data_stored":13380684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":41533891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570137,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55920,"kb_used_data":19168,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94315920,"statfs":{"total":96636764160,"available":96579502080,"internally_reserved":0,"allocated":19628032,"data_stored":15895628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:24:44.990 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:24:45.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:45.109+0000 7f4bcd1f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:45.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:45.125+0000 7f4bcd1f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:45.292 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:24:45.292 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:24:45.309 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1679,"stamp":"2024-03-30T13:24:44.631411+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":297009311,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":448852,"kb_used_data":147792,"kb_used_omap":72,"kb_used_meta":300855,"kb_avail":754525868,"statfs":{"total":773094113280,"available":772634488832,"internally_reserved":0,"allocated":151339008,"data_stored":121500179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74099,"internal_metadata":308076173},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.043042"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1794,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-03-30T13:24:40.104810+0000","last_change":"2024-03-30T13:24:29.448256+0000","last_active":"2024-03-30T13:24:40.104810+0000","last_peered":"2024-03-30T13:24:40.104810+0000","last_clean":"2024-03-30T13:24:40.104810+0000","last_became_active":"2024-03-30T13:24:29.446987+0000","last_became_peered":"2024-03-30T13:24:29.446987+0000","last_unstale":"2024-03-30T13:24:40.104810+0000","last_undegraded":"2024-03-30T13:24:40.104810+0000","last_fullsized":"2024-03-30T13:24:40.104810+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:24:01.591463+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:24:01.591463+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:10.966108+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1683,"reported_epoch":808,"state":"active+clean","last_fresh":"2024-03-30T13:24:42.615283+0000","last_change":"2024-03-30T13:24:29.440117+0000","last_active":"2024-03-30T13:24:42.615283+0000","last_peered":"2024-03-30T13:24:42.615283+0000","last_clean":"2024-03-30T13:24:42.615283+0000","last_became_active":"2024-03-30T13:24:29.438667+0000","last_became_peered":"2024-03-30T13:24:29.438667+0000","last_unstale":"2024-03-30T13:24:42.615283+0000","last_undegraded":"2024-03-30T13:24:42.615283+0000","last_fullsized":"2024-03-30T13:24:42.615283+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:30.947058+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:23:30.947058+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:58.949105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":515,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-03-30T13:24:41.214789+0000","last_change":"2024-03-30T13:24:29.442221+0000","last_active":"2024-03-30T13:24:41.214789+0000","last_peered":"2024-03-30T13:24:41.214789+0000","last_clean":"2024-03-30T13:24:41.214789+0000","last_became_active":"2024-03-30T13:24:29.440460+0000","last_became_peered":"2024-03-30T13:24:29.440460+0000","last_unstale":"2024-03-30T13:24:41.214789+0000","last_undegraded":"2024-03-30T13:24:41.214789+0000","last_fullsized":"2024-03-30T13:24:41.214789+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:24:27.040925+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:24:27.040925+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:51.105436+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":39075780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":481,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-03-30T13:24:41.214693+0000","last_change":"2024-03-30T13:23:36.180822+0000","last_active":"2024-03-30T13:24:41.214693+0000","last_peered":"2024-03-30T13:24:41.214693+0000","last_clean":"2024-03-30T13:24:41.214693+0000","last_became_active":"2024-03-30T13:21:56.541153+0000","last_became_peered":"2024-03-30T13:21:56.541153+0000","last_unstale":"2024-03-30T13:24:41.214693+0000","last_undegraded":"2024-03-30T13:24:41.214693+0000","last_fullsized":"2024-03-30T13:24:41.214693+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:23:36.180557+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:52.909200+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":662,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-03-30T13:24:40.619528+0000","last_change":"2024-03-30T13:24:24.431877+0000","last_active":"2024-03-30T13:24:40.619528+0000","last_peered":"2024-03-30T13:24:40.619528+0000","last_clean":"2024-03-30T13:24:40.619528+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:24:40.619528+0000","last_undegraded":"2024-03-30T13:24:40.619528+0000","last_fullsized":"2024-03-30T13:24:40.619528+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:24:24.431636+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:24:24.431636+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:46.235372+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1724,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-03-30T13:24:40.592747+0000","last_change":"2024-03-30T13:23:40.597999+0000","last_active":"2024-03-30T13:24:40.592747+0000","last_peered":"2024-03-30T13:24:40.592747+0000","last_clean":"2024-03-30T13:24:40.592747+0000","last_became_active":"2024-03-30T13:22:47.491313+0000","last_became_peered":"2024-03-30T13:22:47.491313+0000","last_unstale":"2024-03-30T13:24:40.592747+0000","last_undegraded":"2024-03-30T13:24:40.592747+0000","last_fullsized":"2024-03-30T13:24:40.592747+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:23:40.597839+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:23:40.597839+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:00.377458+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":549,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-03-30T13:24:41.214736+0000","last_change":"2024-03-30T13:24:32.092922+0000","last_active":"2024-03-30T13:24:41.214736+0000","last_peered":"2024-03-30T13:24:41.214736+0000","last_clean":"2024-03-30T13:24:41.214736+0000","last_became_active":"2024-03-30T13:24:29.440149+0000","last_became_peered":"2024-03-30T13:24:29.440149+0000","last_unstale":"2024-03-30T13:24:41.214736+0000","last_undegraded":"2024-03-30T13:24:41.214736+0000","last_fullsized":"2024-03-30T13:24:41.214736+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:24:32.092620+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:24:32.092620+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:42.584351+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":43916097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1095,"reported_epoch":807,"state":"active+clean","last_fresh":"2024-03-30T13:24:41.609310+0000","last_change":"2024-03-30T13:24:35.540725+0000","last_active":"2024-03-30T13:24:41.609310+0000","last_peered":"2024-03-30T13:24:41.609310+0000","last_clean":"2024-03-30T13:24:41.609310+0000","last_became_active":"2024-03-30T13:23:38.514736+0000","last_became_peered":"2024-03-30T13:23:38.514736+0000","last_unstale":"2024-03-30T13:24:41.609310+0000","last_undegraded":"2024-03-30T13:24:41.609310+0000","last_fullsized":"2024-03-30T13:24:41.609310+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:24:35.540394+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:24:35.540394+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:42.188887+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1827,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-03-30T13:24:40.592449+0000","last_change":"2024-03-30T13:23:27.681044+0000","last_active":"2024-03-30T13:24:40.592449+0000","last_peered":"2024-03-30T13:24:40.592449+0000","last_clean":"2024-03-30T13:24:40.592449+0000","last_became_active":"2024-03-30T13:22:47.491356+0000","last_became_peered":"2024-03-30T13:22:47.491356+0000","last_unstale":"2024-03-30T13:24:40.592449+0000","last_undegraded":"2024-03-30T13:24:40.592449+0000","last_fullsized":"2024-03-30T13:24:40.592449+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:23:27.680784+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:45.246430+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1513,"reported_epoch":808,"state":"active+clean","last_fresh":"2024-03-30T13:24:42.615145+0000","last_change":"2024-03-30T13:24:30.839444+0000","last_active":"2024-03-30T13:24:42.615145+0000","last_peered":"2024-03-30T13:24:42.615145+0000","last_clean":"2024-03-30T13:24:42.615145+0000","last_became_active":"2024-03-30T13:24:29.442802+0000","last_became_peered":"2024-03-30T13:24:29.442802+0000","last_unstale":"2024-03-30T13:24:42.615145+0000","last_undegraded":"2024-03-30T13:24:42.615145+0000","last_fullsized":"2024-03-30T13:24:42.615145+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:24:30.839164+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:24:30.839164+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:38.428401+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1819,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-03-30T13:24:40.104367+0000","last_change":"2024-03-30T13:24:29.448451+0000","last_active":"2024-03-30T13:24:40.104367+0000","last_peered":"2024-03-30T13:24:40.104367+0000","last_clean":"2024-03-30T13:24:40.104367+0000","last_became_active":"2024-03-30T13:24:29.447202+0000","last_became_peered":"2024-03-30T13:24:29.447202+0000","last_unstale":"2024-03-30T13:24:40.104367+0000","last_undegraded":"2024-03-30T13:24:40.104367+0000","last_fullsized":"2024-03-30T13:24:40.104367+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:23:19.611935+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:23:19.611935+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:43.473381+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1883,"reported_epoch":805,"state":"active+clean","last_fresh":"2024-03-30T13:24:40.104374+0000","last_change":"2024-03-30T13:23:22.573707+0000","last_active":"2024-03-30T13:24:40.104374+0000","last_peered":"2024-03-30T13:24:40.104374+0000","last_clean":"2024-03-30T13:24:40.104374+0000","last_became_active":"2024-03-30T13:22:47.488417+0000","last_became_peered":"2024-03-30T13:22:47.488417+0000","last_unstale":"2024-03-30T13:24:40.104374+0000","last_undegraded":"2024-03-30T13:24:40.104374+0000","last_fullsized":"2024-03-30T13:24:40.104374+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:23:22.573400+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:43.538364+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":717,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-03-30T13:24:40.619478+0000","last_change":"2024-03-30T13:24:10.389590+0000","last_active":"2024-03-30T13:24:40.619478+0000","last_peered":"2024-03-30T13:24:40.619478+0000","last_clean":"2024-03-30T13:24:40.619478+0000","last_became_active":"2024-03-30T13:22:47.493033+0000","last_became_peered":"2024-03-30T13:22:47.493033+0000","last_unstale":"2024-03-30T13:24:40.619478+0000","last_undegraded":"2024-03-30T13:24:40.619478+0000","last_fullsized":"2024-03-30T13:24:40.619478+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:24:10.389317+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:24:10.389317+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:17.072573+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18319366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":485,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-03-30T13:24:41.214761+0000","last_change":"2024-03-30T13:23:51.071627+0000","last_active":"2024-03-30T13:24:41.214761+0000","last_peered":"2024-03-30T13:24:41.214761+0000","last_clean":"2024-03-30T13:24:41.214761+0000","last_became_active":"2024-03-30T13:22:47.489008+0000","last_became_peered":"2024-03-30T13:22:47.489008+0000","last_unstale":"2024-03-30T13:24:41.214761+0000","last_undegraded":"2024-03-30T13:24:41.214761+0000","last_fullsized":"2024-03-30T13:24:41.214761+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:51.071390+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:23:51.071390+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:10.481199+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4974806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":702,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-03-30T13:24:40.633815+0000","last_change":"2024-03-30T13:24:40.633893+0000","last_active":"2024-03-30T13:24:40.633815+0000","last_peered":"2024-03-30T13:24:40.633815+0000","last_clean":"2024-03-30T13:24:40.633815+0000","last_became_active":"2024-03-30T13:21:56.534725+0000","last_became_peered":"2024-03-30T13:21:56.534725+0000","last_unstale":"2024-03-30T13:24:40.633815+0000","last_undegraded":"2024-03-30T13:24:40.633815+0000","last_fullsized":"2024-03-30T13:24:40.633815+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_clean_scrub_stamp":"2024-03-30T13:24:40.633617+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:52.628929+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":958,"reported_epoch":807,"state":"active+clean","last_fresh":"2024-03-30T13:24:41.609413+0000","last_change":"2024-03-30T13:24:36.495425+0000","last_active":"2024-03-30T13:24:41.609413+0000","last_peered":"2024-03-30T13:24:41.609413+0000","last_clean":"2024-03-30T13:24:41.609413+0000","last_became_active":"2024-03-30T13:23:38.516830+0000","last_became_peered":"2024-03-30T13:23:38.516830+0000","last_unstale":"2024-03-30T13:24:41.609413+0000","last_undegraded":"2024-03-30T13:24:41.609413+0000","last_fullsized":"2024-03-30T13:24:41.609413+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:24:36.495187+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:24:36.495187+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:37.050877+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":297009311,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387747,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58956,"kb_used_data":21752,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94312884,"statfs":{"total":96636764160,"available":96576393216,"internally_reserved":0,"allocated":22274048,"data_stored":18555246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157563,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64176,"kb_used_data":26520,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94307664,"statfs":{"total":96636764160,"available":96571047936,"internally_reserved":0,"allocated":27156480,"data_stored":23424076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":794,"seq":3410204033029,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58772,"kb_used_data":18248,"kb_used_omap":15,"kb_used_meta":40496,"kb_avail":94313068,"statfs":{"total":96636764160,"available":96576581632,"internally_reserved":0,"allocated":18685952,"data_stored":14939946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15941,"internal_metadata":41468347},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509793,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48308,"kb_used_data":12968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323532,"statfs":{"total":96636764160,"available":96587296768,"internally_reserved":0,"allocated":13279232,"data_stored":9560430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":769,"seq":3302829850639,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49932,"kb_used_data":12280,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94321908,"statfs":{"total":96636764160,"available":96585633792,"internally_reserved":0,"allocated":12574720,"data_stored":8835515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10995,"internal_metadata":38524173},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509794,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55484,"kb_used_data":20144,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94316356,"statfs":{"total":96636764160,"available":96579948544,"internally_reserved":0,"allocated":20627456,"data_stored":16908654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742467,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57304,"kb_used_data":16712,"kb_used_omap":15,"kb_used_meta":40560,"kb_avail":94314536,"statfs":{"total":96636764160,"available":96578084864,"internally_reserved":0,"allocated":17113088,"data_stored":13380684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":41533891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570137,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55920,"kb_used_data":19168,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94315920,"statfs":{"total":96636764160,"available":96579502080,"internally_reserved":0,"allocated":19628032,"data_stored":15895628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:24:45.310 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:24:45.310 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:24:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:45.421+0000 7f831e80c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:45.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:45.437+0000 7f831e80c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:45.608 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:24:45.608 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":810,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:24:44.627232+0000","last_up_change":"2024-03-30T13:24:28.423921+0000","last_in_change":"2024-03-30T13:24:28.423921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":39,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":787,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":581,"last_clean_end":763,"up_from":769,"up_thru":769,"down_at":764,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4049884108},{"type":"v1","addr":"172.21.15.67:6818","nonce":4049884108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4049884108},{"type":"v1","addr":"172.21.15.67:6821","nonce":4049884108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":4049884108},{"type":"v1","addr":"172.21.15.67:6828","nonce":4049884108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":4049884108},{"type":"v1","addr":"172.21.15.67:6825","nonce":4049884108}]},"public_addr":"172.21.15.67:6818/4049884108","cluster_addr":"172.21.15.67:6821/4049884108","heartbeat_back_addr":"172.21.15.67:6828/4049884108","heartbeat_front_addr":"172.21.15.67:6825/4049884108","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":787,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":699,"last_clean_end":738,"up_from":747,"up_thru":747,"down_at":739,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6801","nonce":3161027348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6803","nonce":3161027348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6807","nonce":3161027348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6805","nonce":3161027348}]},"public_addr":"172.21.15.67:6801/3161027348","cluster_addr":"172.21.15.67:6803/3161027348","heartbeat_back_addr":"172.21.15.67:6807/3161027348","heartbeat_front_addr":"172.21.15.67:6805/3161027348","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":677,"last_clean_end":785,"up_from":794,"up_thru":794,"down_at":786,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":167623373},{"type":"v1","addr":"172.21.15.82:6803","nonce":167623373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":167623373},{"type":"v1","addr":"172.21.15.82:6807","nonce":167623373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":167623373},{"type":"v1","addr":"172.21.15.82:6815","nonce":167623373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":167623373},{"type":"v1","addr":"172.21.15.82:6811","nonce":167623373}]},"public_addr":"172.21.15.82:6803/167623373","cluster_addr":"172.21.15.82:6807/167623373","heartbeat_back_addr":"172.21.15.82:6815/167623373","heartbeat_front_addr":"172.21.15.82:6811/167623373","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":488,"last_clean_end":506,"up_from":515,"up_thru":747,"down_at":507,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6817","nonce":292505453}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6819","nonce":292505453}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6823","nonce":292505453}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":292505453},{"type":"v1","addr":"172.21.15.82:6821","nonce":292505453}]},"public_addr":"172.21.15.82:6817/292505453","cluster_addr":"172.21.15.82:6819/292505453","heartbeat_back_addr":"172.21.15.82:6823/292505453","heartbeat_front_addr":"172.21.15.82:6821/292505453","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":629,"last_clean_end":715,"up_from":722,"up_thru":794,"down_at":716,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6801","nonce":326680821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6806","nonce":326680821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6814","nonce":326680821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6810","nonce":326680821}]},"public_addr":"172.21.15.82:6801/326680821","cluster_addr":"172.21.15.82:6806/326680821","heartbeat_back_addr":"172.21.15.82:6814/326680821","heartbeat_front_addr":"172.21.15.82:6810/326680821","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":604,"last_clean_end":645,"up_from":650,"up_thru":794,"down_at":646,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6825","nonce":1497758647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6827","nonce":1497758647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6831","nonce":1497758647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6829","nonce":1497758647}]},"public_addr":"172.21.15.82:6825/1497758647","cluster_addr":"172.21.15.82:6827/1497758647","heartbeat_back_addr":"172.21.15.82:6831/1497758647","heartbeat_front_addr":"172.21.15.82:6829/1497758647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:24:14.902038+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:23:44.725062+0000","dead_epoch":764},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:23:46.060234+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:24:07.848049+0000","dead_epoch":739},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:24:33.848220+0000","dead_epoch":786},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:24:05.245546+0000","dead_epoch":507},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:23:30.766774+0000","dead_epoch":716},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:23:58.512646+0000","dead_epoch":646}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:24:45.627 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:24:45.627 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:24:45.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:45.741+0000 7f3c858d3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:45.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:45.761+0000 7f3c858d3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:45.931 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:24:45.950 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:24:46.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:46.065+0000 7f1c8972b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:46.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:46.081+0000 7f1c8972b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:46.260 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:24:46.261 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:24:46.279 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:24:46.279 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:24:46.279 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:24:46.279 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:24:46.279 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 5, 7, 6, 3, 1, 4] 2024-03-30T13:24:46.279 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:24:46.280 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:24:46.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:46.397+0000 7fd3673ca640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:46.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:46.413+0000 7fd3673ca640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:46.584 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:24:46.584 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:24:46.602 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1680,"stamp":"2024-03-30T13:24:46.292538+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":297009311,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":449048,"kb_used_data":147972,"kb_used_omap":72,"kb_used_meta":300855,"kb_avail":754525672,"statfs":{"total":773094113280,"available":772634288128,"internally_reserved":0,"allocated":151523328,"data_stored":121582484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74099,"internal_metadata":308076173},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.687413"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1802,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-03-30T13:24:43.744656+0000","last_change":"2024-03-30T13:24:29.448256+0000","last_active":"2024-03-30T13:24:43.744656+0000","last_peered":"2024-03-30T13:24:43.744656+0000","last_clean":"2024-03-30T13:24:43.744656+0000","last_became_active":"2024-03-30T13:24:29.446987+0000","last_became_peered":"2024-03-30T13:24:29.446987+0000","last_unstale":"2024-03-30T13:24:43.744656+0000","last_undegraded":"2024-03-30T13:24:43.744656+0000","last_fullsized":"2024-03-30T13:24:43.744656+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:24:01.591463+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:24:01.591463+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:10.966108+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1683,"reported_epoch":808,"state":"active+clean","last_fresh":"2024-03-30T13:24:42.615283+0000","last_change":"2024-03-30T13:24:29.440117+0000","last_active":"2024-03-30T13:24:42.615283+0000","last_peered":"2024-03-30T13:24:42.615283+0000","last_clean":"2024-03-30T13:24:42.615283+0000","last_became_active":"2024-03-30T13:24:29.438667+0000","last_became_peered":"2024-03-30T13:24:29.438667+0000","last_unstale":"2024-03-30T13:24:42.615283+0000","last_undegraded":"2024-03-30T13:24:42.615283+0000","last_fullsized":"2024-03-30T13:24:42.615283+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:30.947058+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:23:30.947058+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:58.949105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":515,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-03-30T13:24:41.214789+0000","last_change":"2024-03-30T13:24:29.442221+0000","last_active":"2024-03-30T13:24:41.214789+0000","last_peered":"2024-03-30T13:24:41.214789+0000","last_clean":"2024-03-30T13:24:41.214789+0000","last_became_active":"2024-03-30T13:24:29.440460+0000","last_became_peered":"2024-03-30T13:24:29.440460+0000","last_unstale":"2024-03-30T13:24:41.214789+0000","last_undegraded":"2024-03-30T13:24:41.214789+0000","last_fullsized":"2024-03-30T13:24:41.214789+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:24:27.040925+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:24:27.040925+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:51.105436+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":39075780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":481,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-03-30T13:24:41.214693+0000","last_change":"2024-03-30T13:23:36.180822+0000","last_active":"2024-03-30T13:24:41.214693+0000","last_peered":"2024-03-30T13:24:41.214693+0000","last_clean":"2024-03-30T13:24:41.214693+0000","last_became_active":"2024-03-30T13:21:56.541153+0000","last_became_peered":"2024-03-30T13:21:56.541153+0000","last_unstale":"2024-03-30T13:24:41.214693+0000","last_undegraded":"2024-03-30T13:24:41.214693+0000","last_fullsized":"2024-03-30T13:24:41.214693+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:23:36.180557+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:52.909200+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":662,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-03-30T13:24:40.619528+0000","last_change":"2024-03-30T13:24:24.431877+0000","last_active":"2024-03-30T13:24:40.619528+0000","last_peered":"2024-03-30T13:24:40.619528+0000","last_clean":"2024-03-30T13:24:40.619528+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:24:40.619528+0000","last_undegraded":"2024-03-30T13:24:40.619528+0000","last_fullsized":"2024-03-30T13:24:40.619528+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:24:24.431636+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:24:24.431636+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:46.235372+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1724,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-03-30T13:24:40.592747+0000","last_change":"2024-03-30T13:23:40.597999+0000","last_active":"2024-03-30T13:24:40.592747+0000","last_peered":"2024-03-30T13:24:40.592747+0000","last_clean":"2024-03-30T13:24:40.592747+0000","last_became_active":"2024-03-30T13:22:47.491313+0000","last_became_peered":"2024-03-30T13:22:47.491313+0000","last_unstale":"2024-03-30T13:24:40.592747+0000","last_undegraded":"2024-03-30T13:24:40.592747+0000","last_fullsized":"2024-03-30T13:24:40.592747+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:23:40.597839+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:23:40.597839+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:00.377458+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":549,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-03-30T13:24:41.214736+0000","last_change":"2024-03-30T13:24:32.092922+0000","last_active":"2024-03-30T13:24:41.214736+0000","last_peered":"2024-03-30T13:24:41.214736+0000","last_clean":"2024-03-30T13:24:41.214736+0000","last_became_active":"2024-03-30T13:24:29.440149+0000","last_became_peered":"2024-03-30T13:24:29.440149+0000","last_unstale":"2024-03-30T13:24:41.214736+0000","last_undegraded":"2024-03-30T13:24:41.214736+0000","last_fullsized":"2024-03-30T13:24:41.214736+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:24:32.092620+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:24:32.092620+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:42.584351+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":43916097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1095,"reported_epoch":807,"state":"active+clean","last_fresh":"2024-03-30T13:24:41.609310+0000","last_change":"2024-03-30T13:24:35.540725+0000","last_active":"2024-03-30T13:24:41.609310+0000","last_peered":"2024-03-30T13:24:41.609310+0000","last_clean":"2024-03-30T13:24:41.609310+0000","last_became_active":"2024-03-30T13:23:38.514736+0000","last_became_peered":"2024-03-30T13:23:38.514736+0000","last_unstale":"2024-03-30T13:24:41.609310+0000","last_undegraded":"2024-03-30T13:24:41.609310+0000","last_fullsized":"2024-03-30T13:24:41.609310+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:24:35.540394+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:24:35.540394+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:42.188887+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1827,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-03-30T13:24:40.592449+0000","last_change":"2024-03-30T13:23:27.681044+0000","last_active":"2024-03-30T13:24:40.592449+0000","last_peered":"2024-03-30T13:24:40.592449+0000","last_clean":"2024-03-30T13:24:40.592449+0000","last_became_active":"2024-03-30T13:22:47.491356+0000","last_became_peered":"2024-03-30T13:22:47.491356+0000","last_unstale":"2024-03-30T13:24:40.592449+0000","last_undegraded":"2024-03-30T13:24:40.592449+0000","last_fullsized":"2024-03-30T13:24:40.592449+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:23:27.680784+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:45.246430+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1513,"reported_epoch":808,"state":"active+clean","last_fresh":"2024-03-30T13:24:42.615145+0000","last_change":"2024-03-30T13:24:30.839444+0000","last_active":"2024-03-30T13:24:42.615145+0000","last_peered":"2024-03-30T13:24:42.615145+0000","last_clean":"2024-03-30T13:24:42.615145+0000","last_became_active":"2024-03-30T13:24:29.442802+0000","last_became_peered":"2024-03-30T13:24:29.442802+0000","last_unstale":"2024-03-30T13:24:42.615145+0000","last_undegraded":"2024-03-30T13:24:42.615145+0000","last_fullsized":"2024-03-30T13:24:42.615145+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:24:30.839164+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:24:30.839164+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:38.428401+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1835,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-03-30T13:24:43.915535+0000","last_change":"2024-03-30T13:24:43.915621+0000","last_active":"2024-03-30T13:24:43.915535+0000","last_peered":"2024-03-30T13:24:43.915535+0000","last_clean":"2024-03-30T13:24:43.915535+0000","last_became_active":"2024-03-30T13:24:29.447202+0000","last_became_peered":"2024-03-30T13:24:29.447202+0000","last_unstale":"2024-03-30T13:24:43.915535+0000","last_undegraded":"2024-03-30T13:24:43.915535+0000","last_fullsized":"2024-03-30T13:24:43.915535+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:24:43.915348+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:24:43.915348+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:58.654177+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1891,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-03-30T13:24:43.744607+0000","last_change":"2024-03-30T13:23:22.573707+0000","last_active":"2024-03-30T13:24:43.744607+0000","last_peered":"2024-03-30T13:24:43.744607+0000","last_clean":"2024-03-30T13:24:43.744607+0000","last_became_active":"2024-03-30T13:22:47.488417+0000","last_became_peered":"2024-03-30T13:22:47.488417+0000","last_unstale":"2024-03-30T13:24:43.744607+0000","last_undegraded":"2024-03-30T13:24:43.744607+0000","last_fullsized":"2024-03-30T13:24:43.744607+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:23:22.573400+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":717,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-03-30T13:24:40.619478+0000","last_change":"2024-03-30T13:24:10.389590+0000","last_active":"2024-03-30T13:24:40.619478+0000","last_peered":"2024-03-30T13:24:40.619478+0000","last_clean":"2024-03-30T13:24:40.619478+0000","last_became_active":"2024-03-30T13:22:47.493033+0000","last_became_peered":"2024-03-30T13:22:47.493033+0000","last_unstale":"2024-03-30T13:24:40.619478+0000","last_undegraded":"2024-03-30T13:24:40.619478+0000","last_fullsized":"2024-03-30T13:24:40.619478+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:24:10.389317+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:24:10.389317+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:17.072573+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18319366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":485,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-03-30T13:24:41.214761+0000","last_change":"2024-03-30T13:23:51.071627+0000","last_active":"2024-03-30T13:24:41.214761+0000","last_peered":"2024-03-30T13:24:41.214761+0000","last_clean":"2024-03-30T13:24:41.214761+0000","last_became_active":"2024-03-30T13:22:47.489008+0000","last_became_peered":"2024-03-30T13:22:47.489008+0000","last_unstale":"2024-03-30T13:24:41.214761+0000","last_undegraded":"2024-03-30T13:24:41.214761+0000","last_fullsized":"2024-03-30T13:24:41.214761+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:51.071390+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:23:51.071390+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:10.481199+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4974806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":702,"reported_epoch":806,"state":"active+clean","last_fresh":"2024-03-30T13:24:40.633815+0000","last_change":"2024-03-30T13:24:40.633893+0000","last_active":"2024-03-30T13:24:40.633815+0000","last_peered":"2024-03-30T13:24:40.633815+0000","last_clean":"2024-03-30T13:24:40.633815+0000","last_became_active":"2024-03-30T13:21:56.534725+0000","last_became_peered":"2024-03-30T13:21:56.534725+0000","last_unstale":"2024-03-30T13:24:40.633815+0000","last_undegraded":"2024-03-30T13:24:40.633815+0000","last_fullsized":"2024-03-30T13:24:40.633815+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_clean_scrub_stamp":"2024-03-30T13:24:40.633617+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:52.628929+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":958,"reported_epoch":807,"state":"active+clean","last_fresh":"2024-03-30T13:24:41.609413+0000","last_change":"2024-03-30T13:24:36.495425+0000","last_active":"2024-03-30T13:24:41.609413+0000","last_peered":"2024-03-30T13:24:41.609413+0000","last_clean":"2024-03-30T13:24:41.609413+0000","last_became_active":"2024-03-30T13:23:38.516830+0000","last_became_peered":"2024-03-30T13:23:38.516830+0000","last_unstale":"2024-03-30T13:24:41.609413+0000","last_undegraded":"2024-03-30T13:24:41.609413+0000","last_fullsized":"2024-03-30T13:24:41.609413+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:24:36.495187+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:24:36.495187+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:37.050877+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":297009311,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387748,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59016,"kb_used_data":21812,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94312824,"statfs":{"total":96636764160,"available":96576331776,"internally_reserved":0,"allocated":22335488,"data_stored":18582681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":515,"seq":2211908157563,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64176,"kb_used_data":26520,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94307664,"statfs":{"total":96636764160,"available":96571047936,"internally_reserved":0,"allocated":27156480,"data_stored":23424076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11000,"internal_metadata":38524168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":794,"seq":3410204033029,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58772,"kb_used_data":18248,"kb_used_omap":15,"kb_used_meta":40496,"kb_avail":94313068,"statfs":{"total":96636764160,"available":96576581632,"internally_reserved":0,"allocated":18685952,"data_stored":14939946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15941,"internal_metadata":41468347},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509794,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48376,"kb_used_data":13028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323464,"statfs":{"total":96636764160,"available":96587227136,"internally_reserved":0,"allocated":13340672,"data_stored":9587865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":769,"seq":3302829850639,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49932,"kb_used_data":12280,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94321908,"statfs":{"total":96636764160,"available":96585633792,"internally_reserved":0,"allocated":12574720,"data_stored":8835515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10995,"internal_metadata":38524173},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509795,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55552,"kb_used_data":20204,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94316288,"statfs":{"total":96636764160,"available":96579878912,"internally_reserved":0,"allocated":20688896,"data_stored":16936089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742467,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57304,"kb_used_data":16712,"kb_used_omap":15,"kb_used_meta":40560,"kb_avail":94314536,"statfs":{"total":96636764160,"available":96578084864,"internally_reserved":0,"allocated":17113088,"data_stored":13380684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":41533891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570137,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55920,"kb_used_data":19168,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94315920,"statfs":{"total":96636764160,"available":96579502080,"internally_reserved":0,"allocated":19628032,"data_stored":15895628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:24:46.603 INFO:tasks.thrashosds.thrasher:Choosing PG 1.a with acting set [5, 2, 6] 2024-03-30T13:24:46.603 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 2, 5, 7, 6, 3, 1, 4] 2024-03-30T13:24:46.603 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-30T13:24:46.604 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:24:46.675 INFO:tasks.ceph.osd.5:Stopped 2024-03-30T13:24:46.676 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 2, 5, 7, 6, 3, 1, 4] 2024-03-30T13:24:46.676 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 5 2024-03-30T13:24:46.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:46.797+0000 7fe9da5ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:46.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:46.813+0000 7fe9da5ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:46.995 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:24:46.997+0000 7fbb38f08640 -1 mon.a@0(leader).osd e810 definitely_dead 0 2024-03-30T13:24:47.650 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.5. 2024-03-30T13:24:47.683 INFO:tasks.thrashosds.thrasher:dead_osds=[5], live_osds=[0, 2, 7, 6, 3, 1, 4] 2024-03-30T13:24:47.683 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:24:47.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:24:47.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:47.801+0000 7fca32065640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:47.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:24:47.813+0000 7fca32065640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:24:47.986 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:24:47.986 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:24:48.004 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1681,"stamp":"2024-03-30T13:24:47.653929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":297009311,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":385000,"kb_used_data":121584,"kb_used_omap":61,"kb_used_meta":263234,"kb_avail":660217880,"statfs":{"total":676457349120,"available":676063109120,"internally_reserved":0,"allocated":124502016,"data_stored":98218765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63099,"internal_metadata":269552005},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.362422"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1802,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-03-30T13:24:43.744656+0000","last_change":"2024-03-30T13:24:29.448256+0000","last_active":"2024-03-30T13:24:43.744656+0000","last_peered":"2024-03-30T13:24:43.744656+0000","last_clean":"2024-03-30T13:24:43.744656+0000","last_became_active":"2024-03-30T13:24:29.446987+0000","last_became_peered":"2024-03-30T13:24:29.446987+0000","last_unstale":"2024-03-30T13:24:43.744656+0000","last_undegraded":"2024-03-30T13:24:43.744656+0000","last_fullsized":"2024-03-30T13:24:43.744656+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:24:01.591463+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:24:01.591463+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:10.966108+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9176490,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1683,"reported_epoch":808,"state":"active+clean","last_fresh":"2024-03-30T13:24:42.615283+0000","last_change":"2024-03-30T13:24:29.440117+0000","last_active":"2024-03-30T13:24:42.615283+0000","last_peered":"2024-03-30T13:24:42.615283+0000","last_clean":"2024-03-30T13:24:42.615283+0000","last_became_active":"2024-03-30T13:24:29.438667+0000","last_became_peered":"2024-03-30T13:24:29.438667+0000","last_unstale":"2024-03-30T13:24:42.615283+0000","last_undegraded":"2024-03-30T13:24:42.615283+0000","last_fullsized":"2024-03-30T13:24:42.615283+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:30.947058+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:23:30.947058+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:58.949105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":523,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-03-30T13:24:46.142377+0000","last_change":"2024-03-30T13:24:29.442221+0000","last_active":"2024-03-30T13:24:46.142377+0000","last_peered":"2024-03-30T13:24:46.142377+0000","last_clean":"2024-03-30T13:24:46.142377+0000","last_became_active":"2024-03-30T13:24:29.440460+0000","last_became_peered":"2024-03-30T13:24:29.440460+0000","last_unstale":"2024-03-30T13:24:46.142377+0000","last_undegraded":"2024-03-30T13:24:46.142377+0000","last_fullsized":"2024-03-30T13:24:46.142377+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:24:27.040925+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:24:27.040925+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:51.105436+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":39075780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":489,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-03-30T13:24:46.142549+0000","last_change":"2024-03-30T13:23:36.180822+0000","last_active":"2024-03-30T13:24:46.142549+0000","last_peered":"2024-03-30T13:24:46.142549+0000","last_clean":"2024-03-30T13:24:46.142549+0000","last_became_active":"2024-03-30T13:21:56.541153+0000","last_became_peered":"2024-03-30T13:21:56.541153+0000","last_unstale":"2024-03-30T13:24:46.142549+0000","last_undegraded":"2024-03-30T13:24:46.142549+0000","last_fullsized":"2024-03-30T13:24:46.142549+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:23:36.180557+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:24:52.909200+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":662,"reported_epoch":806,"state":"stale+active+clean","last_fresh":"2024-03-30T13:24:40.619528+0000","last_change":"2024-03-30T13:24:24.431877+0000","last_active":"2024-03-30T13:24:40.619528+0000","last_peered":"2024-03-30T13:24:40.619528+0000","last_clean":"2024-03-30T13:24:40.619528+0000","last_became_active":"2024-03-30T13:14:38.233688+0000","last_became_peered":"2024-03-30T13:14:38.233688+0000","last_unstale":"2024-03-30T13:24:47.653954+0000","last_undegraded":"2024-03-30T13:24:40.619528+0000","last_fullsized":"2024-03-30T13:24:40.619528+0000","mapping_epoch":515,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":516,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:24:24.431636+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:24:24.431636+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:46.235372+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1732,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-03-30T13:24:44.636445+0000","last_change":"2024-03-30T13:23:40.597999+0000","last_active":"2024-03-30T13:24:44.636445+0000","last_peered":"2024-03-30T13:24:44.636445+0000","last_clean":"2024-03-30T13:24:44.636445+0000","last_became_active":"2024-03-30T13:22:47.491313+0000","last_became_peered":"2024-03-30T13:22:47.491313+0000","last_unstale":"2024-03-30T13:24:44.636445+0000","last_undegraded":"2024-03-30T13:24:44.636445+0000","last_fullsized":"2024-03-30T13:24:44.636445+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:23:40.597839+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:23:40.597839+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:00.377458+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":557,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-03-30T13:24:46.142408+0000","last_change":"2024-03-30T13:24:32.092922+0000","last_active":"2024-03-30T13:24:46.142408+0000","last_peered":"2024-03-30T13:24:46.142408+0000","last_clean":"2024-03-30T13:24:46.142408+0000","last_became_active":"2024-03-30T13:24:29.440149+0000","last_became_peered":"2024-03-30T13:24:29.440149+0000","last_unstale":"2024-03-30T13:24:46.142408+0000","last_undegraded":"2024-03-30T13:24:46.142408+0000","last_fullsized":"2024-03-30T13:24:46.142408+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:24:32.092620+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:24:32.092620+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:42.584351+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":43916097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1101,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-03-30T13:24:46.359520+0000","last_change":"2024-03-30T13:24:35.540725+0000","last_active":"2024-03-30T13:24:46.359520+0000","last_peered":"2024-03-30T13:24:46.359520+0000","last_clean":"2024-03-30T13:24:46.359520+0000","last_became_active":"2024-03-30T13:23:38.514736+0000","last_became_peered":"2024-03-30T13:23:38.514736+0000","last_unstale":"2024-03-30T13:24:46.359520+0000","last_undegraded":"2024-03-30T13:24:46.359520+0000","last_fullsized":"2024-03-30T13:24:46.359520+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:24:35.540394+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:24:35.540394+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:42.188887+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1843,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-03-30T13:24:46.148310+0000","last_change":"2024-03-30T13:24:46.148416+0000","last_active":"2024-03-30T13:24:46.148310+0000","last_peered":"2024-03-30T13:24:46.148310+0000","last_clean":"2024-03-30T13:24:46.148310+0000","last_became_active":"2024-03-30T13:22:47.491356+0000","last_became_peered":"2024-03-30T13:22:47.491356+0000","last_unstale":"2024-03-30T13:24:46.148310+0000","last_undegraded":"2024-03-30T13:24:46.148310+0000","last_fullsized":"2024-03-30T13:24:46.148310+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:24:46.148128+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:24:46.148128+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:14.365782+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":60907776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1513,"reported_epoch":808,"state":"active+clean","last_fresh":"2024-03-30T13:24:42.615145+0000","last_change":"2024-03-30T13:24:30.839444+0000","last_active":"2024-03-30T13:24:42.615145+0000","last_peered":"2024-03-30T13:24:42.615145+0000","last_clean":"2024-03-30T13:24:42.615145+0000","last_became_active":"2024-03-30T13:24:29.442802+0000","last_became_peered":"2024-03-30T13:24:29.442802+0000","last_unstale":"2024-03-30T13:24:42.615145+0000","last_undegraded":"2024-03-30T13:24:42.615145+0000","last_fullsized":"2024-03-30T13:24:42.615145+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:24:30.839164+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:24:30.839164+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:38.428401+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1835,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-03-30T13:24:43.915535+0000","last_change":"2024-03-30T13:24:43.915621+0000","last_active":"2024-03-30T13:24:43.915535+0000","last_peered":"2024-03-30T13:24:43.915535+0000","last_clean":"2024-03-30T13:24:43.915535+0000","last_became_active":"2024-03-30T13:24:29.447202+0000","last_became_peered":"2024-03-30T13:24:29.447202+0000","last_unstale":"2024-03-30T13:24:43.915535+0000","last_undegraded":"2024-03-30T13:24:43.915535+0000","last_fullsized":"2024-03-30T13:24:43.915535+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:24:43.915348+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:24:43.915348+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:58.654177+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1891,"reported_epoch":809,"state":"active+clean","last_fresh":"2024-03-30T13:24:43.744607+0000","last_change":"2024-03-30T13:23:22.573707+0000","last_active":"2024-03-30T13:24:43.744607+0000","last_peered":"2024-03-30T13:24:43.744607+0000","last_clean":"2024-03-30T13:24:43.744607+0000","last_became_active":"2024-03-30T13:22:47.488417+0000","last_became_peered":"2024-03-30T13:22:47.488417+0000","last_unstale":"2024-03-30T13:24:43.744607+0000","last_undegraded":"2024-03-30T13:24:43.744607+0000","last_fullsized":"2024-03-30T13:24:43.744607+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:23:22.573400+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":19243751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":717,"reported_epoch":806,"state":"stale+active+clean","last_fresh":"2024-03-30T13:24:40.619478+0000","last_change":"2024-03-30T13:24:10.389590+0000","last_active":"2024-03-30T13:24:40.619478+0000","last_peered":"2024-03-30T13:24:40.619478+0000","last_clean":"2024-03-30T13:24:40.619478+0000","last_became_active":"2024-03-30T13:22:47.493033+0000","last_became_peered":"2024-03-30T13:22:47.493033+0000","last_unstale":"2024-03-30T13:24:47.653958+0000","last_undegraded":"2024-03-30T13:24:40.619478+0000","last_fullsized":"2024-03-30T13:24:40.619478+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:24:10.389317+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:24:10.389317+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:17.072573+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18319366,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":493,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-03-30T13:24:46.142576+0000","last_change":"2024-03-30T13:23:51.071627+0000","last_active":"2024-03-30T13:24:46.142576+0000","last_peered":"2024-03-30T13:24:46.142576+0000","last_clean":"2024-03-30T13:24:46.142576+0000","last_became_active":"2024-03-30T13:22:47.489008+0000","last_became_peered":"2024-03-30T13:22:47.489008+0000","last_unstale":"2024-03-30T13:24:46.142576+0000","last_undegraded":"2024-03-30T13:24:46.142576+0000","last_fullsized":"2024-03-30T13:24:46.142576+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:51.071390+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:23:51.071390+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:10.481199+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4974806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":702,"reported_epoch":806,"state":"stale+active+clean","last_fresh":"2024-03-30T13:24:40.633815+0000","last_change":"2024-03-30T13:24:40.633893+0000","last_active":"2024-03-30T13:24:40.633815+0000","last_peered":"2024-03-30T13:24:40.633815+0000","last_clean":"2024-03-30T13:24:40.633815+0000","last_became_active":"2024-03-30T13:21:56.534725+0000","last_became_peered":"2024-03-30T13:21:56.534725+0000","last_unstale":"2024-03-30T13:24:47.653963+0000","last_undegraded":"2024-03-30T13:24:40.633815+0000","last_fullsized":"2024-03-30T13:24:40.633815+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_clean_scrub_stamp":"2024-03-30T13:24:40.633617+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:52.628929+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":18151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":964,"reported_epoch":810,"state":"active+clean","last_fresh":"2024-03-30T13:24:46.359545+0000","last_change":"2024-03-30T13:24:36.495425+0000","last_active":"2024-03-30T13:24:46.359545+0000","last_peered":"2024-03-30T13:24:46.359545+0000","last_clean":"2024-03-30T13:24:46.359545+0000","last_became_active":"2024-03-30T13:23:38.516830+0000","last_became_peered":"2024-03-30T13:23:38.516830+0000","last_unstale":"2024-03-30T13:24:46.359545+0000","last_undegraded":"2024-03-30T13:24:46.359545+0000","last_fullsized":"2024-03-30T13:24:46.359545+0000","mapping_epoch":769,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":770,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:24:36.495187+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:24:36.495187+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:37.050877+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":297009311,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387748,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59016,"kb_used_data":21812,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94312824,"statfs":{"total":96636764160,"available":96576331776,"internally_reserved":0,"allocated":22335488,"data_stored":18582681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":794,"seq":3410204033029,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58772,"kb_used_data":18248,"kb_used_omap":15,"kb_used_meta":40496,"kb_avail":94313068,"statfs":{"total":96636764160,"available":96576581632,"internally_reserved":0,"allocated":18685952,"data_stored":14939946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15941,"internal_metadata":41468347},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509794,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48376,"kb_used_data":13028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94323464,"statfs":{"total":96636764160,"available":96587227136,"internally_reserved":0,"allocated":13340672,"data_stored":9587865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":769,"seq":3302829850640,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49960,"kb_used_data":12316,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94321880,"statfs":{"total":96636764160,"available":96585605120,"internally_reserved":0,"allocated":12611584,"data_stored":8851976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10995,"internal_metadata":38524173},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509795,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55552,"kb_used_data":20204,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94316288,"statfs":{"total":96636764160,"available":96579878912,"internally_reserved":0,"allocated":20688896,"data_stored":16936089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742468,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57348,"kb_used_data":16760,"kb_used_omap":15,"kb_used_meta":40560,"kb_avail":94314492,"statfs":{"total":96636764160,"available":96578039808,"internally_reserved":0,"allocated":17162240,"data_stored":13402632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":41533891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570138,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55976,"kb_used_data":19216,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94315864,"statfs":{"total":96636764160,"available":96579444736,"internally_reserved":0,"allocated":19677184,"data_stored":15917576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:24:48.006 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:25:13.007 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:25:13.007 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:25:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:13.136+0000 7f623d9f8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:13.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:13.148+0000 7f623d9f8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:13.323 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:25:13.323 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:25:13.341 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1697,"stamp":"2024-03-30T13:25:12.300238+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":333883374,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":409300,"kb_used_data":141760,"kb_used_omap":61,"kb_used_meta":267394,"kb_avail":660193580,"statfs":{"total":676457349120,"available":676038225920,"internally_reserved":0,"allocated":145162240,"data_stored":118697945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":63099,"internal_metadata":273811845},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003097"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1822,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-03-30T13:24:57.429358+0000","last_change":"2024-03-30T13:24:48.816534+0000","last_active":"2024-03-30T13:24:57.429358+0000","last_peered":"2024-03-30T13:24:57.429358+0000","last_clean":"2024-03-30T13:24:57.429358+0000","last_became_active":"2024-03-30T13:24:48.682275+0000","last_became_peered":"2024-03-30T13:24:48.682275+0000","last_unstale":"2024-03-30T13:24:57.429358+0000","last_undegraded":"2024-03-30T13:24:57.429358+0000","last_fullsized":"2024-03-30T13:24:57.429358+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:24:01.591463+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:53.569165+0000","last_clean_scrub_stamp":"2024-03-30T13:24:01.591463+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:11.344191+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":13764735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1701,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-03-30T13:24:59.916037+0000","last_change":"2024-03-30T13:24:59.916106+0000","last_active":"2024-03-30T13:24:59.916037+0000","last_peered":"2024-03-30T13:24:59.916037+0000","last_clean":"2024-03-30T13:24:59.916037+0000","last_became_active":"2024-03-30T13:24:29.438667+0000","last_became_peered":"2024-03-30T13:24:29.438667+0000","last_unstale":"2024-03-30T13:24:59.916037+0000","last_undegraded":"2024-03-30T13:24:59.916037+0000","last_fullsized":"2024-03-30T13:24:59.916037+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:24:59.915857+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:24:59.915857+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:12.958686+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":529,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-03-30T13:24:58.204553+0000","last_change":"2024-03-30T13:24:29.442221+0000","last_active":"2024-03-30T13:24:58.204553+0000","last_peered":"2024-03-30T13:24:58.204553+0000","last_clean":"2024-03-30T13:24:58.204553+0000","last_became_active":"2024-03-30T13:24:29.440460+0000","last_became_peered":"2024-03-30T13:24:29.440460+0000","last_unstale":"2024-03-30T13:24:58.204553+0000","last_undegraded":"2024-03-30T13:24:58.204553+0000","last_fullsized":"2024-03-30T13:24:58.204553+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:24:27.040925+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:24:27.040925+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:51.105436+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":39075780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":503,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-03-30T13:24:58.204166+0000","last_change":"2024-03-30T13:24:53.286146+0000","last_active":"2024-03-30T13:24:58.204166+0000","last_peered":"2024-03-30T13:24:58.204166+0000","last_clean":"2024-03-30T13:24:58.204166+0000","last_became_active":"2024-03-30T13:21:56.541153+0000","last_became_peered":"2024-03-30T13:21:56.541153+0000","last_unstale":"2024-03-30T13:24:58.204166+0000","last_undegraded":"2024-03-30T13:24:58.204166+0000","last_fullsized":"2024-03-30T13:24:58.204166+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:24:53.285743+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:24:53.285743+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:10.430742+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":17,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-03-30T13:24:58.203873+0000","last_change":"2024-03-30T13:24:48.779710+0000","last_active":"2024-03-30T13:24:58.203873+0000","last_peered":"2024-03-30T13:24:58.203873+0000","last_clean":"2024-03-30T13:24:58.203873+0000","last_became_active":"2024-03-30T13:24:48.677788+0000","last_became_peered":"2024-03-30T13:24:48.677788+0000","last_unstale":"2024-03-30T13:24:58.203873+0000","last_undegraded":"2024-03-30T13:24:58.203873+0000","last_fullsized":"2024-03-30T13:24:58.203873+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:24:24.431636+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:24:24.431636+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:32.661533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2235298,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1767,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-03-30T13:25:11.485521+0000","last_change":"2024-03-30T13:25:01.341609+0000","last_active":"2024-03-30T13:25:11.485521+0000","last_peered":"2024-03-30T13:25:11.485521+0000","last_clean":"2024-03-30T13:25:11.485521+0000","last_became_active":"2024-03-30T13:24:48.673043+0000","last_became_peered":"2024-03-30T13:24:48.673043+0000","last_unstale":"2024-03-30T13:25:11.485521+0000","last_undegraded":"2024-03-30T13:25:11.485521+0000","last_fullsized":"2024-03-30T13:25:11.485521+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:25:01.341334+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:25:01.341334+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:22.312671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11394981,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":563,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-03-30T13:24:58.204317+0000","last_change":"2024-03-30T13:24:32.092922+0000","last_active":"2024-03-30T13:24:58.204317+0000","last_peered":"2024-03-30T13:24:58.204317+0000","last_clean":"2024-03-30T13:24:58.204317+0000","last_became_active":"2024-03-30T13:24:29.440149+0000","last_became_peered":"2024-03-30T13:24:29.440149+0000","last_unstale":"2024-03-30T13:24:58.204317+0000","last_undegraded":"2024-03-30T13:24:58.204317+0000","last_fullsized":"2024-03-30T13:24:58.204317+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:24:32.092620+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:24:32.092620+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:42.584351+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":43916097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1121,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-03-30T13:24:57.431295+0000","last_change":"2024-03-30T13:24:48.738408+0000","last_active":"2024-03-30T13:24:57.431295+0000","last_peered":"2024-03-30T13:24:57.431295+0000","last_clean":"2024-03-30T13:24:57.431295+0000","last_became_active":"2024-03-30T13:24:48.671806+0000","last_became_peered":"2024-03-30T13:24:48.671806+0000","last_unstale":"2024-03-30T13:24:57.431295+0000","last_undegraded":"2024-03-30T13:24:57.431295+0000","last_fullsized":"2024-03-30T13:24:57.431295+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:24:35.540394+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:24:35.540394+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:03.195134+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1864,"reported_epoch":814,"state":"active+clean","last_fresh":"2024-03-30T13:25:11.485195+0000","last_change":"2024-03-30T13:24:49.499129+0000","last_active":"2024-03-30T13:25:11.485195+0000","last_peered":"2024-03-30T13:25:11.485195+0000","last_clean":"2024-03-30T13:25:11.485195+0000","last_became_active":"2024-03-30T13:24:49.455289+0000","last_became_peered":"2024-03-30T13:24:49.455289+0000","last_unstale":"2024-03-30T13:25:11.485195+0000","last_undegraded":"2024-03-30T13:25:11.485195+0000","last_fullsized":"2024-03-30T13:25:11.485195+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:24:46.148128+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:24:46.148128+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:14.679653+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":65983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1523,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-03-30T13:24:58.204250+0000","last_change":"2024-03-30T13:24:30.839444+0000","last_active":"2024-03-30T13:24:58.204250+0000","last_peered":"2024-03-30T13:24:58.204250+0000","last_clean":"2024-03-30T13:24:58.204250+0000","last_became_active":"2024-03-30T13:24:29.442802+0000","last_became_peered":"2024-03-30T13:24:29.442802+0000","last_unstale":"2024-03-30T13:24:58.204250+0000","last_undegraded":"2024-03-30T13:24:58.204250+0000","last_fullsized":"2024-03-30T13:24:58.204250+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:24:30.839164+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:24:30.839164+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:38.428401+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1843,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-03-30T13:24:57.429211+0000","last_change":"2024-03-30T13:24:43.915621+0000","last_active":"2024-03-30T13:24:57.429211+0000","last_peered":"2024-03-30T13:24:57.429211+0000","last_clean":"2024-03-30T13:24:57.429211+0000","last_became_active":"2024-03-30T13:24:29.447202+0000","last_became_peered":"2024-03-30T13:24:29.447202+0000","last_unstale":"2024-03-30T13:24:57.429211+0000","last_undegraded":"2024-03-30T13:24:57.429211+0000","last_fullsized":"2024-03-30T13:24:57.429211+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:24:43.915348+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:24:43.915348+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:58.654177+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1931,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-03-30T13:24:57.429359+0000","last_change":"2024-03-30T13:24:56.750994+0000","last_active":"2024-03-30T13:24:57.429359+0000","last_peered":"2024-03-30T13:24:57.429359+0000","last_clean":"2024-03-30T13:24:57.429359+0000","last_became_active":"2024-03-30T13:24:48.674847+0000","last_became_peered":"2024-03-30T13:24:48.674847+0000","last_unstale":"2024-03-30T13:24:57.429359+0000","last_undegraded":"2024-03-30T13:24:57.429359+0000","last_fullsized":"2024-03-30T13:24:57.429359+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:24:56.750718+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:24:56.750718+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:26.348937+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":23,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-03-30T13:24:57.429498+0000","last_change":"2024-03-30T13:24:48.939780+0000","last_active":"2024-03-30T13:24:57.429498+0000","last_peered":"2024-03-30T13:24:57.429498+0000","last_clean":"2024-03-30T13:24:57.429498+0000","last_became_active":"2024-03-30T13:24:48.683105+0000","last_became_peered":"2024-03-30T13:24:48.683105+0000","last_unstale":"2024-03-30T13:24:57.429498+0000","last_undegraded":"2024-03-30T13:24:57.429498+0000","last_fullsized":"2024-03-30T13:24:57.429498+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:24:10.389317+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:24:10.389317+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:14.737314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9159683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":499,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-03-30T13:24:58.204193+0000","last_change":"2024-03-30T13:23:51.071627+0000","last_active":"2024-03-30T13:24:58.204193+0000","last_peered":"2024-03-30T13:24:58.204193+0000","last_clean":"2024-03-30T13:24:58.204193+0000","last_became_active":"2024-03-30T13:22:47.489008+0000","last_became_peered":"2024-03-30T13:22:47.489008+0000","last_unstale":"2024-03-30T13:24:58.204193+0000","last_undegraded":"2024-03-30T13:24:58.204193+0000","last_fullsized":"2024-03-30T13:24:58.204193+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:23:51.071390+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:23:51.071390+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:10.481199+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4974806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":23,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-03-30T13:24:58.203923+0000","last_change":"2024-03-30T13:24:49.437713+0000","last_active":"2024-03-30T13:24:58.203923+0000","last_peered":"2024-03-30T13:24:58.203923+0000","last_clean":"2024-03-30T13:24:58.203923+0000","last_became_active":"2024-03-30T13:24:48.679257+0000","last_became_peered":"2024-03-30T13:24:48.679257+0000","last_unstale":"2024-03-30T13:24:58.203923+0000","last_undegraded":"2024-03-30T13:24:58.203923+0000","last_fullsized":"2024-03-30T13:24:58.203923+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_clean_scrub_stamp":"2024-03-30T13:24:40.633617+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:09.887060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":9075648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":982,"reported_epoch":813,"state":"active+clean","last_fresh":"2024-03-30T13:24:57.431347+0000","last_change":"2024-03-30T13:24:48.855829+0000","last_active":"2024-03-30T13:24:57.431347+0000","last_peered":"2024-03-30T13:24:57.431347+0000","last_clean":"2024-03-30T13:24:57.431347+0000","last_became_active":"2024-03-30T13:24:48.684509+0000","last_became_peered":"2024-03-30T13:24:48.684509+0000","last_unstale":"2024-03-30T13:24:57.431347+0000","last_undegraded":"2024-03-30T13:24:57.431347+0000","last_fullsized":"2024-03-30T13:24:57.431347+0000","mapping_epoch":811,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":812,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:24:36.495187+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:24:36.495187+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:40.785438+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5226911,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136,"num_bytes_recovered":333883374,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":117080064,"data_stored":117080064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387753,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64620,"kb_used_data":27424,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94307220,"statfs":{"total":96636764160,"available":96570593280,"internally_reserved":0,"allocated":28082176,"data_stored":24302528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":794,"seq":3410204033034,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63328,"kb_used_data":22804,"kb_used_omap":15,"kb_used_meta":40496,"kb_avail":94308512,"statfs":{"total":96636764160,"available":96571916288,"internally_reserved":0,"allocated":23351296,"data_stored":19571648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15941,"internal_metadata":41468347},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509799,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55204,"kb_used_data":15704,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94316636,"statfs":{"total":96636764160,"available":96580235264,"internally_reserved":0,"allocated":16080896,"data_stored":12301248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":769,"seq":3302829850644,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55208,"kb_used_data":17564,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94316632,"statfs":{"total":96636764160,"available":96580231168,"internally_reserved":0,"allocated":17985536,"data_stored":14205888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10995,"internal_metadata":38524173},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509800,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55592,"kb_used_data":20252,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94316248,"statfs":{"total":96636764160,"available":96579837952,"internally_reserved":0,"allocated":20738048,"data_stored":16958400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742472,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57384,"kb_used_data":16796,"kb_used_omap":15,"kb_used_meta":40560,"kb_avail":94314456,"statfs":{"total":96636764160,"available":96578002944,"internally_reserved":0,"allocated":17199104,"data_stored":13419456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":41533891},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570143,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57964,"kb_used_data":21216,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94313876,"statfs":{"total":96636764160,"available":96577409024,"internally_reserved":0,"allocated":21725184,"data_stored":17938777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":11071488,"data_stored":11071488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9166848,"data_stored":9166848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":14798848,"data_stored":14798848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16437248,"data_stored":16437248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":21168128,"data_stored":21168128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:25:13.342 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:25:13.342 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:25:13.343 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-03-30T13:25:13.343 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-30T13:25:13.343 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-30T13:25:13.345 INFO:tasks.ceph.osd.5:Started 2024-03-30T13:25:13.346 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-30T13:25:13.414 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:25:13.413+0000 7f61345be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:13.422 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:25:13.421+0000 7f61345be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:13.422 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:25:13.421+0000 7f61345be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:13.473 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T13:25:13.473 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T13:25:13.474 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T13:25:13.474 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T13:25:13.474 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T13:25:13.474 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T13:25:13.474 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-03-30T13:25:13.474 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-03-30T13:25:13.474 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-03-30T13:25:13.474 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-03-30T13:25:13.474 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-03-30T13:25:13.474 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T13:25:13.475 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:25:13.476 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-03-30T13:25:13.958 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:25:13.957+0000 7f61345be7c0 -1 Falling back to public interface 2024-03-30T13:25:17.766 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:25:17.765+0000 7f61345be7c0 -1 osd.5 810 log_to_monitors true 2024-03-30T13:25:18.477 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-03-30T13:25:18.545 INFO:tasks.ceph.osd.5.smithi082.stderr:2024-03-30T13:25:18.545+0000 7f612c393640 -1 osd.5 810 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:25:18.611 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T13:25:18.611 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T13:25:18.611 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T13:25:18.611 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T13:25:18.625 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:25:18.625 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-03-30T13:25:18.625 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 5 2024-03-30T13:25:18.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:18.752+0000 7f6aa4a31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:18.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:18.768+0000 7f6aa4a31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:18.955 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:25:18.956+0000 7fbb38f08640 -1 mon.a@0(leader).osd e815 definitely_dead 0 2024-03-30T13:25:19.518 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.5. 2024-03-30T13:25:19.553 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-03-30T13:25:19.553 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:25:19.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:19.668+0000 7f6f73b15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:19.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:19.684+0000 7f6f73b15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:20.544 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:25:20.574 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:25:20.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:20.692+0000 7efc96545640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:20.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:20.704+0000 7efc96545640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:21.547 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:25:21.572 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:25:21.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:21.688+0000 7f1032a94640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:21.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:21.704+0000 7f1032a94640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:22.561 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:25:22.590 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:25:22.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:22.712+0000 7faed1230640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:22.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:22.728+0000 7faed1230640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:23.597 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:25:23.629 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:25:23.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:23.748+0000 7fd6f76ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:23.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:23.768+0000 7fd6f76ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:24.599 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:25:24.622 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:25:24.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:24.740+0000 7feb9d244640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:24.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:24.756+0000 7feb9d244640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:25.602 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:25:25.626 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:25:25.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:25.744+0000 7fb210f24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:25.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:25.760+0000 7fb210f24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:26.617 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:25:26.640 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:25:26.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:26.760+0000 7f37d116c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:26.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:26.776+0000 7f37d116c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:27.618 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:25:27.645 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:25:27.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:27.764+0000 7f354b0c9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:27.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:27.780+0000 7f354b0c9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:28.624 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:25:28.649 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:25:28.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:28.768+0000 7f7fbe615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:28.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:28.788+0000 7f7fbe615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:29.634 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:25:29.662 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:25:29.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:29.780+0000 7fbce9bed640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:29.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:29.796+0000 7fbce9bed640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:30.644 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:25:30.669 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:25:30.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:30.788+0000 7fd3ddeea640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:30.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:30.804+0000 7fd3ddeea640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:31.655 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:25:31.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:25:31.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:31.803+0000 7fc376160640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:31.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:31.819+0000 7fc376160640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:32.665 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:25:32.690 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:25:32.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:32.799+0000 7f8b3e80f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:32.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:32.815+0000 7f8b3e80f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:33.692 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:25:33.714 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:25:33.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:33.831+0000 7f153533f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:33.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:33.847+0000 7f153533f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:34.690 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:25:34.715 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:25:34.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:34.831+0000 7f077ba12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:34.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:34.847+0000 7f077ba12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:35.699 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:25:35.726 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:25:35.727 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:25:35.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:35.843+0000 7f59a49f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:35.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:35.859+0000 7f59a49f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:36.032 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:25:36.032 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:25:36.050 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1726,"stamp":"2024-03-30T13:25:35.702911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":313412745,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":459328,"kb_used_data":149964,"kb_used_omap":73,"kb_used_meta":309174,"kb_avail":754515392,"statfs":{"total":773094113280,"available":772623761408,"internally_reserved":0,"allocated":153563136,"data_stored":122510806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":75753,"internal_metadata":316594199},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.043079"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1868,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-30T13:25:34.999668+0000","last_change":"2024-03-30T13:25:20.550328+0000","last_active":"2024-03-30T13:25:34.999668+0000","last_peered":"2024-03-30T13:25:34.999668+0000","last_clean":"2024-03-30T13:25:34.999668+0000","last_became_active":"2024-03-30T13:25:20.549084+0000","last_became_peered":"2024-03-30T13:25:20.549084+0000","last_unstale":"2024-03-30T13:25:34.999668+0000","last_undegraded":"2024-03-30T13:25:34.999668+0000","last_fullsized":"2024-03-30T13:25:34.999668+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:25:12.543076+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:25:12.543076+0000","last_clean_scrub_stamp":"2024-03-30T13:25:12.543076+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:32.027663+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":13764735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1728,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-03-30T13:25:30.650008+0000","last_change":"2024-03-30T13:24:59.916106+0000","last_active":"2024-03-30T13:25:30.650008+0000","last_peered":"2024-03-30T13:25:30.650008+0000","last_clean":"2024-03-30T13:25:30.650008+0000","last_became_active":"2024-03-30T13:24:29.438667+0000","last_became_peered":"2024-03-30T13:24:29.438667+0000","last_unstale":"2024-03-30T13:25:30.650008+0000","last_undegraded":"2024-03-30T13:25:30.650008+0000","last_fullsized":"2024-03-30T13:25:30.650008+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:24:59.915857+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:24:59.915857+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:12.958686+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":554,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-03-30T13:25:30.294336+0000","last_change":"2024-03-30T13:24:29.442221+0000","last_active":"2024-03-30T13:25:30.294336+0000","last_peered":"2024-03-30T13:25:30.294336+0000","last_clean":"2024-03-30T13:25:30.294336+0000","last_became_active":"2024-03-30T13:24:29.440460+0000","last_became_peered":"2024-03-30T13:24:29.440460+0000","last_unstale":"2024-03-30T13:25:30.294336+0000","last_undegraded":"2024-03-30T13:25:30.294336+0000","last_fullsized":"2024-03-30T13:25:30.294336+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:24:27.040925+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:24:27.040925+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:51.105436+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":39075780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":529,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-03-30T13:25:30.294021+0000","last_change":"2024-03-30T13:24:53.286146+0000","last_active":"2024-03-30T13:25:30.294021+0000","last_peered":"2024-03-30T13:25:30.294021+0000","last_clean":"2024-03-30T13:25:30.294021+0000","last_became_active":"2024-03-30T13:21:56.541153+0000","last_became_peered":"2024-03-30T13:21:56.541153+0000","last_unstale":"2024-03-30T13:25:30.294021+0000","last_undegraded":"2024-03-30T13:25:30.294021+0000","last_fullsized":"2024-03-30T13:25:30.294021+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:24:53.285743+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:24:53.285743+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:10.430742+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":668,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-03-30T13:25:34.211450+0000","last_change":"2024-03-30T13:25:20.557379+0000","last_active":"2024-03-30T13:25:34.211450+0000","last_peered":"2024-03-30T13:25:34.211450+0000","last_clean":"2024-03-30T13:25:34.211450+0000","last_became_active":"2024-03-30T13:25:20.556165+0000","last_became_peered":"2024-03-30T13:25:20.556165+0000","last_unstale":"2024-03-30T13:25:34.211450+0000","last_undegraded":"2024-03-30T13:25:34.211450+0000","last_fullsized":"2024-03-30T13:25:34.211450+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:24:24.431636+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:24:24.431636+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:47.484058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1798,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-03-30T13:25:30.651324+0000","last_change":"2024-03-30T13:25:20.552405+0000","last_active":"2024-03-30T13:25:30.651324+0000","last_peered":"2024-03-30T13:25:30.651324+0000","last_clean":"2024-03-30T13:25:30.651324+0000","last_became_active":"2024-03-30T13:25:20.551941+0000","last_became_peered":"2024-03-30T13:25:20.551941+0000","last_unstale":"2024-03-30T13:25:30.651324+0000","last_undegraded":"2024-03-30T13:25:30.651324+0000","last_fullsized":"2024-03-30T13:25:30.651324+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:25:01.341334+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:25:01.341334+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:21.421274+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11394981,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":588,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-03-30T13:25:30.294080+0000","last_change":"2024-03-30T13:24:32.092922+0000","last_active":"2024-03-30T13:25:30.294080+0000","last_peered":"2024-03-30T13:25:30.294080+0000","last_clean":"2024-03-30T13:25:30.294080+0000","last_became_active":"2024-03-30T13:24:29.440149+0000","last_became_peered":"2024-03-30T13:24:29.440149+0000","last_unstale":"2024-03-30T13:25:30.294080+0000","last_undegraded":"2024-03-30T13:25:30.294080+0000","last_fullsized":"2024-03-30T13:25:30.294080+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:24:32.092620+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:24:32.092620+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:42.584351+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":43916097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1155,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-03-30T13:25:31.994865+0000","last_change":"2024-03-30T13:25:20.551362+0000","last_active":"2024-03-30T13:25:31.994865+0000","last_peered":"2024-03-30T13:25:31.994865+0000","last_clean":"2024-03-30T13:25:31.994865+0000","last_became_active":"2024-03-30T13:25:20.549997+0000","last_became_peered":"2024-03-30T13:25:20.549997+0000","last_unstale":"2024-03-30T13:25:31.994865+0000","last_undegraded":"2024-03-30T13:25:31.994865+0000","last_fullsized":"2024-03-30T13:25:31.994865+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:24:35.540394+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:24:35.540394+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:48.570974+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1895,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-03-30T13:25:30.651289+0000","last_change":"2024-03-30T13:25:20.553343+0000","last_active":"2024-03-30T13:25:30.651289+0000","last_peered":"2024-03-30T13:25:30.651289+0000","last_clean":"2024-03-30T13:25:30.651289+0000","last_became_active":"2024-03-30T13:25:20.552377+0000","last_became_peered":"2024-03-30T13:25:20.552377+0000","last_unstale":"2024-03-30T13:25:30.651289+0000","last_undegraded":"2024-03-30T13:25:30.651289+0000","last_fullsized":"2024-03-30T13:25:30.651289+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:24:46.148128+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:24:46.148128+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:10.093152+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":65983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1550,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-03-30T13:25:30.649863+0000","last_change":"2024-03-30T13:24:30.839444+0000","last_active":"2024-03-30T13:25:30.649863+0000","last_peered":"2024-03-30T13:25:30.649863+0000","last_clean":"2024-03-30T13:25:30.649863+0000","last_became_active":"2024-03-30T13:24:29.442802+0000","last_became_peered":"2024-03-30T13:24:29.442802+0000","last_unstale":"2024-03-30T13:25:30.649863+0000","last_undegraded":"2024-03-30T13:25:30.649863+0000","last_fullsized":"2024-03-30T13:25:30.649863+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:24:30.839164+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:24:30.839164+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:38.428401+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1875,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-30T13:25:34.999565+0000","last_change":"2024-03-30T13:24:43.915621+0000","last_active":"2024-03-30T13:25:34.999565+0000","last_peered":"2024-03-30T13:25:34.999565+0000","last_clean":"2024-03-30T13:25:34.999565+0000","last_became_active":"2024-03-30T13:24:29.447202+0000","last_became_peered":"2024-03-30T13:24:29.447202+0000","last_unstale":"2024-03-30T13:25:34.999565+0000","last_undegraded":"2024-03-30T13:25:34.999565+0000","last_fullsized":"2024-03-30T13:25:34.999565+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:24:43.915348+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:24:43.915348+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:58.654177+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1969,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-30T13:25:34.999536+0000","last_change":"2024-03-30T13:25:20.551031+0000","last_active":"2024-03-30T13:25:34.999536+0000","last_peered":"2024-03-30T13:25:34.999536+0000","last_clean":"2024-03-30T13:25:34.999536+0000","last_became_active":"2024-03-30T13:25:20.549465+0000","last_became_peered":"2024-03-30T13:25:20.549465+0000","last_unstale":"2024-03-30T13:25:34.999536+0000","last_undegraded":"2024-03-30T13:25:34.999536+0000","last_fullsized":"2024-03-30T13:25:34.999536+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:24:56.750718+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:24:56.750718+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:09.232296+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":721,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-03-30T13:25:34.211642+0000","last_change":"2024-03-30T13:25:20.551619+0000","last_active":"2024-03-30T13:25:34.211642+0000","last_peered":"2024-03-30T13:25:34.211642+0000","last_clean":"2024-03-30T13:25:34.211642+0000","last_became_active":"2024-03-30T13:25:20.549427+0000","last_became_peered":"2024-03-30T13:25:20.549427+0000","last_unstale":"2024-03-30T13:25:34.211642+0000","last_undegraded":"2024-03-30T13:25:34.211642+0000","last_fullsized":"2024-03-30T13:25:34.211642+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:25:15.484555+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:25:15.484555+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:34.070118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":532,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-03-30T13:25:30.294420+0000","last_change":"2024-03-30T13:25:12.391849+0000","last_active":"2024-03-30T13:25:30.294420+0000","last_peered":"2024-03-30T13:25:30.294420+0000","last_clean":"2024-03-30T13:25:30.294420+0000","last_became_active":"2024-03-30T13:22:47.489008+0000","last_became_peered":"2024-03-30T13:22:47.489008+0000","last_unstale":"2024-03-30T13:25:30.294420+0000","last_undegraded":"2024-03-30T13:25:30.294420+0000","last_fullsized":"2024-03-30T13:25:30.294420+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:25:12.391479+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:25:12.391479+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:26.494439+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4974806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":736,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-03-30T13:25:34.211382+0000","last_change":"2024-03-30T13:25:20.558649+0000","last_active":"2024-03-30T13:25:34.211382+0000","last_peered":"2024-03-30T13:25:34.211382+0000","last_clean":"2024-03-30T13:25:34.211382+0000","last_became_active":"2024-03-30T13:25:20.556235+0000","last_became_peered":"2024-03-30T13:25:20.556235+0000","last_unstale":"2024-03-30T13:25:34.211382+0000","last_undegraded":"2024-03-30T13:25:34.211382+0000","last_fullsized":"2024-03-30T13:25:34.211382+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_clean_scrub_stamp":"2024-03-30T13:24:40.633617+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:08.997018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":1016,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-03-30T13:25:31.994960+0000","last_change":"2024-03-30T13:25:20.552502+0000","last_active":"2024-03-30T13:25:31.994960+0000","last_peered":"2024-03-30T13:25:31.994960+0000","last_clean":"2024-03-30T13:25:31.994960+0000","last_became_active":"2024-03-30T13:25:20.550231+0000","last_became_peered":"2024-03-30T13:25:20.550231+0000","last_unstale":"2024-03-30T13:25:31.994960+0000","last_undegraded":"2024-03-30T13:25:31.994960+0000","last_fullsized":"2024-03-30T13:25:31.994960+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:24:36.495187+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:24:36.495187+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:58.381669+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5226911,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":313412745,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387758,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59260,"kb_used_data":22064,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94312580,"statfs":{"total":96636764160,"available":96576081920,"internally_reserved":0,"allocated":22593536,"data_stored":18700093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":816,"seq":3504693313541,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68608,"kb_used_data":26796,"kb_used_omap":12,"kb_used_meta":41779,"kb_avail":94303232,"statfs":{"total":96636764160,"available":96566509568,"internally_reserved":0,"allocated":27439104,"data_stored":23552462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12654,"internal_metadata":42782354},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":794,"seq":3410204033039,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59000,"kb_used_data":18476,"kb_used_omap":15,"kb_used_meta":40496,"kb_avail":94312840,"statfs":{"total":96636764160,"available":96576348160,"internally_reserved":0,"allocated":18919424,"data_stored":15046384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15941,"internal_metadata":41468347},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509804,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52784,"kb_used_data":13280,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94319056,"statfs":{"total":96636764160,"available":96582713344,"internally_reserved":0,"allocated":13598720,"data_stored":9705277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":769,"seq":3302829850649,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50176,"kb_used_data":12532,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94321664,"statfs":{"total":96636764160,"available":96585383936,"internally_reserved":0,"allocated":12832768,"data_stored":8952927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10995,"internal_metadata":38524173},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509805,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55796,"kb_used_data":20456,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94316044,"statfs":{"total":96636764160,"available":96579629056,"internally_reserved":0,"allocated":20946944,"data_stored":17053501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742477,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57548,"kb_used_data":16952,"kb_used_omap":15,"kb_used_meta":40560,"kb_avail":94314292,"statfs":{"total":96636764160,"available":96577835008,"internally_reserved":0,"allocated":17358848,"data_stored":13492609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":41533891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570147,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56156,"kb_used_data":19408,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94315684,"statfs":{"total":96636764160,"available":96579260416,"internally_reserved":0,"allocated":19873792,"data_stored":16007553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:25:36.051 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:25:36.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:36.163+0000 7f2f00889640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:36.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:36.183+0000 7f2f00889640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:36.357 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:25:36.358 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:25:36.376 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1727,"stamp":"2024-03-30T13:25:36.307107+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":313412745,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":459328,"kb_used_data":149964,"kb_used_omap":73,"kb_used_meta":309174,"kb_avail":754515392,"statfs":{"total":773094113280,"available":772623761408,"internally_reserved":0,"allocated":153563136,"data_stored":122510806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":75753,"internal_metadata":316594199},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001202"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1868,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-30T13:25:34.999668+0000","last_change":"2024-03-30T13:25:20.550328+0000","last_active":"2024-03-30T13:25:34.999668+0000","last_peered":"2024-03-30T13:25:34.999668+0000","last_clean":"2024-03-30T13:25:34.999668+0000","last_became_active":"2024-03-30T13:25:20.549084+0000","last_became_peered":"2024-03-30T13:25:20.549084+0000","last_unstale":"2024-03-30T13:25:34.999668+0000","last_undegraded":"2024-03-30T13:25:34.999668+0000","last_fullsized":"2024-03-30T13:25:34.999668+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:25:12.543076+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:25:12.543076+0000","last_clean_scrub_stamp":"2024-03-30T13:25:12.543076+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:32.027663+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":13764735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1728,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-03-30T13:25:30.650008+0000","last_change":"2024-03-30T13:24:59.916106+0000","last_active":"2024-03-30T13:25:30.650008+0000","last_peered":"2024-03-30T13:25:30.650008+0000","last_clean":"2024-03-30T13:25:30.650008+0000","last_became_active":"2024-03-30T13:24:29.438667+0000","last_became_peered":"2024-03-30T13:24:29.438667+0000","last_unstale":"2024-03-30T13:25:30.650008+0000","last_undegraded":"2024-03-30T13:25:30.650008+0000","last_fullsized":"2024-03-30T13:25:30.650008+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:24:59.915857+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:24:59.915857+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:12.958686+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":554,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-03-30T13:25:30.294336+0000","last_change":"2024-03-30T13:24:29.442221+0000","last_active":"2024-03-30T13:25:30.294336+0000","last_peered":"2024-03-30T13:25:30.294336+0000","last_clean":"2024-03-30T13:25:30.294336+0000","last_became_active":"2024-03-30T13:24:29.440460+0000","last_became_peered":"2024-03-30T13:24:29.440460+0000","last_unstale":"2024-03-30T13:25:30.294336+0000","last_undegraded":"2024-03-30T13:25:30.294336+0000","last_fullsized":"2024-03-30T13:25:30.294336+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:24:27.040925+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:24:27.040925+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:51.105436+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":39075780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":529,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-03-30T13:25:30.294021+0000","last_change":"2024-03-30T13:24:53.286146+0000","last_active":"2024-03-30T13:25:30.294021+0000","last_peered":"2024-03-30T13:25:30.294021+0000","last_clean":"2024-03-30T13:25:30.294021+0000","last_became_active":"2024-03-30T13:21:56.541153+0000","last_became_peered":"2024-03-30T13:21:56.541153+0000","last_unstale":"2024-03-30T13:25:30.294021+0000","last_undegraded":"2024-03-30T13:25:30.294021+0000","last_fullsized":"2024-03-30T13:25:30.294021+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:24:53.285743+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:24:53.285743+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:10.430742+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":668,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-03-30T13:25:34.211450+0000","last_change":"2024-03-30T13:25:20.557379+0000","last_active":"2024-03-30T13:25:34.211450+0000","last_peered":"2024-03-30T13:25:34.211450+0000","last_clean":"2024-03-30T13:25:34.211450+0000","last_became_active":"2024-03-30T13:25:20.556165+0000","last_became_peered":"2024-03-30T13:25:20.556165+0000","last_unstale":"2024-03-30T13:25:34.211450+0000","last_undegraded":"2024-03-30T13:25:34.211450+0000","last_fullsized":"2024-03-30T13:25:34.211450+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:24:24.431636+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:24:24.431636+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:47.484058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1798,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-03-30T13:25:30.651324+0000","last_change":"2024-03-30T13:25:20.552405+0000","last_active":"2024-03-30T13:25:30.651324+0000","last_peered":"2024-03-30T13:25:30.651324+0000","last_clean":"2024-03-30T13:25:30.651324+0000","last_became_active":"2024-03-30T13:25:20.551941+0000","last_became_peered":"2024-03-30T13:25:20.551941+0000","last_unstale":"2024-03-30T13:25:30.651324+0000","last_undegraded":"2024-03-30T13:25:30.651324+0000","last_fullsized":"2024-03-30T13:25:30.651324+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:25:01.341334+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:25:01.341334+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:21.421274+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11394981,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":588,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-03-30T13:25:30.294080+0000","last_change":"2024-03-30T13:24:32.092922+0000","last_active":"2024-03-30T13:25:30.294080+0000","last_peered":"2024-03-30T13:25:30.294080+0000","last_clean":"2024-03-30T13:25:30.294080+0000","last_became_active":"2024-03-30T13:24:29.440149+0000","last_became_peered":"2024-03-30T13:24:29.440149+0000","last_unstale":"2024-03-30T13:25:30.294080+0000","last_undegraded":"2024-03-30T13:25:30.294080+0000","last_fullsized":"2024-03-30T13:25:30.294080+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:24:32.092620+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:24:32.092620+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:42.584351+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":43916097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1155,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-03-30T13:25:31.994865+0000","last_change":"2024-03-30T13:25:20.551362+0000","last_active":"2024-03-30T13:25:31.994865+0000","last_peered":"2024-03-30T13:25:31.994865+0000","last_clean":"2024-03-30T13:25:31.994865+0000","last_became_active":"2024-03-30T13:25:20.549997+0000","last_became_peered":"2024-03-30T13:25:20.549997+0000","last_unstale":"2024-03-30T13:25:31.994865+0000","last_undegraded":"2024-03-30T13:25:31.994865+0000","last_fullsized":"2024-03-30T13:25:31.994865+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:24:35.540394+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:24:35.540394+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:48.570974+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1895,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-03-30T13:25:30.651289+0000","last_change":"2024-03-30T13:25:20.553343+0000","last_active":"2024-03-30T13:25:30.651289+0000","last_peered":"2024-03-30T13:25:30.651289+0000","last_clean":"2024-03-30T13:25:30.651289+0000","last_became_active":"2024-03-30T13:25:20.552377+0000","last_became_peered":"2024-03-30T13:25:20.552377+0000","last_unstale":"2024-03-30T13:25:30.651289+0000","last_undegraded":"2024-03-30T13:25:30.651289+0000","last_fullsized":"2024-03-30T13:25:30.651289+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:24:46.148128+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:24:46.148128+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:10.093152+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":65983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1550,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-03-30T13:25:30.649863+0000","last_change":"2024-03-30T13:24:30.839444+0000","last_active":"2024-03-30T13:25:30.649863+0000","last_peered":"2024-03-30T13:25:30.649863+0000","last_clean":"2024-03-30T13:25:30.649863+0000","last_became_active":"2024-03-30T13:24:29.442802+0000","last_became_peered":"2024-03-30T13:24:29.442802+0000","last_unstale":"2024-03-30T13:25:30.649863+0000","last_undegraded":"2024-03-30T13:25:30.649863+0000","last_fullsized":"2024-03-30T13:25:30.649863+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:24:30.839164+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:24:30.839164+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:38.428401+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1875,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-30T13:25:34.999565+0000","last_change":"2024-03-30T13:24:43.915621+0000","last_active":"2024-03-30T13:25:34.999565+0000","last_peered":"2024-03-30T13:25:34.999565+0000","last_clean":"2024-03-30T13:25:34.999565+0000","last_became_active":"2024-03-30T13:24:29.447202+0000","last_became_peered":"2024-03-30T13:24:29.447202+0000","last_unstale":"2024-03-30T13:25:34.999565+0000","last_undegraded":"2024-03-30T13:25:34.999565+0000","last_fullsized":"2024-03-30T13:25:34.999565+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:24:43.915348+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:24:43.915348+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:58.654177+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1969,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-30T13:25:34.999536+0000","last_change":"2024-03-30T13:25:20.551031+0000","last_active":"2024-03-30T13:25:34.999536+0000","last_peered":"2024-03-30T13:25:34.999536+0000","last_clean":"2024-03-30T13:25:34.999536+0000","last_became_active":"2024-03-30T13:25:20.549465+0000","last_became_peered":"2024-03-30T13:25:20.549465+0000","last_unstale":"2024-03-30T13:25:34.999536+0000","last_undegraded":"2024-03-30T13:25:34.999536+0000","last_fullsized":"2024-03-30T13:25:34.999536+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:24:56.750718+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:24:56.750718+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:09.232296+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":721,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-03-30T13:25:34.211642+0000","last_change":"2024-03-30T13:25:20.551619+0000","last_active":"2024-03-30T13:25:34.211642+0000","last_peered":"2024-03-30T13:25:34.211642+0000","last_clean":"2024-03-30T13:25:34.211642+0000","last_became_active":"2024-03-30T13:25:20.549427+0000","last_became_peered":"2024-03-30T13:25:20.549427+0000","last_unstale":"2024-03-30T13:25:34.211642+0000","last_undegraded":"2024-03-30T13:25:34.211642+0000","last_fullsized":"2024-03-30T13:25:34.211642+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:25:15.484555+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:25:15.484555+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:34.070118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":532,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-03-30T13:25:30.294420+0000","last_change":"2024-03-30T13:25:12.391849+0000","last_active":"2024-03-30T13:25:30.294420+0000","last_peered":"2024-03-30T13:25:30.294420+0000","last_clean":"2024-03-30T13:25:30.294420+0000","last_became_active":"2024-03-30T13:22:47.489008+0000","last_became_peered":"2024-03-30T13:22:47.489008+0000","last_unstale":"2024-03-30T13:25:30.294420+0000","last_undegraded":"2024-03-30T13:25:30.294420+0000","last_fullsized":"2024-03-30T13:25:30.294420+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:25:12.391479+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:25:12.391479+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:26.494439+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4974806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":736,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-03-30T13:25:34.211382+0000","last_change":"2024-03-30T13:25:20.558649+0000","last_active":"2024-03-30T13:25:34.211382+0000","last_peered":"2024-03-30T13:25:34.211382+0000","last_clean":"2024-03-30T13:25:34.211382+0000","last_became_active":"2024-03-30T13:25:20.556235+0000","last_became_peered":"2024-03-30T13:25:20.556235+0000","last_unstale":"2024-03-30T13:25:34.211382+0000","last_undegraded":"2024-03-30T13:25:34.211382+0000","last_fullsized":"2024-03-30T13:25:34.211382+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_clean_scrub_stamp":"2024-03-30T13:24:40.633617+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:08.997018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":1016,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-03-30T13:25:31.994960+0000","last_change":"2024-03-30T13:25:20.552502+0000","last_active":"2024-03-30T13:25:31.994960+0000","last_peered":"2024-03-30T13:25:31.994960+0000","last_clean":"2024-03-30T13:25:31.994960+0000","last_became_active":"2024-03-30T13:25:20.550231+0000","last_became_peered":"2024-03-30T13:25:20.550231+0000","last_unstale":"2024-03-30T13:25:31.994960+0000","last_undegraded":"2024-03-30T13:25:31.994960+0000","last_fullsized":"2024-03-30T13:25:31.994960+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:24:36.495187+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:24:36.495187+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:58.381669+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5226911,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":313412745,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387758,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59260,"kb_used_data":22064,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94312580,"statfs":{"total":96636764160,"available":96576081920,"internally_reserved":0,"allocated":22593536,"data_stored":18700093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":816,"seq":3504693313541,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68608,"kb_used_data":26796,"kb_used_omap":12,"kb_used_meta":41779,"kb_avail":94303232,"statfs":{"total":96636764160,"available":96566509568,"internally_reserved":0,"allocated":27439104,"data_stored":23552462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12654,"internal_metadata":42782354},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":794,"seq":3410204033039,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59000,"kb_used_data":18476,"kb_used_omap":15,"kb_used_meta":40496,"kb_avail":94312840,"statfs":{"total":96636764160,"available":96576348160,"internally_reserved":0,"allocated":18919424,"data_stored":15046384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15941,"internal_metadata":41468347},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509804,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52784,"kb_used_data":13280,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94319056,"statfs":{"total":96636764160,"available":96582713344,"internally_reserved":0,"allocated":13598720,"data_stored":9705277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":769,"seq":3302829850649,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50176,"kb_used_data":12532,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94321664,"statfs":{"total":96636764160,"available":96585383936,"internally_reserved":0,"allocated":12832768,"data_stored":8952927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10995,"internal_metadata":38524173},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509805,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55796,"kb_used_data":20456,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94316044,"statfs":{"total":96636764160,"available":96579629056,"internally_reserved":0,"allocated":20946944,"data_stored":17053501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742477,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57548,"kb_used_data":16952,"kb_used_omap":15,"kb_used_meta":40560,"kb_avail":94314292,"statfs":{"total":96636764160,"available":96577835008,"internally_reserved":0,"allocated":17358848,"data_stored":13492609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":41533891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570147,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56156,"kb_used_data":19408,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94315684,"statfs":{"total":96636764160,"available":96579260416,"internally_reserved":0,"allocated":19873792,"data_stored":16007553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:25:36.377 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:25:36.377 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:25:36.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:36.495+0000 7f48fb5ba640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:36.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:36.507+0000 7f48fb5ba640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:36.686 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:25:36.686 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":832,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:25:35.699977+0000","last_up_change":"2024-03-30T13:25:19.518314+0000","last_in_change":"2024-03-30T13:25:19.518314+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":40,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":787,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":581,"last_clean_end":763,"up_from":769,"up_thru":816,"down_at":764,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4049884108},{"type":"v1","addr":"172.21.15.67:6818","nonce":4049884108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4049884108},{"type":"v1","addr":"172.21.15.67:6821","nonce":4049884108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":4049884108},{"type":"v1","addr":"172.21.15.67:6828","nonce":4049884108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":4049884108},{"type":"v1","addr":"172.21.15.67:6825","nonce":4049884108}]},"public_addr":"172.21.15.67:6818/4049884108","cluster_addr":"172.21.15.67:6821/4049884108","heartbeat_back_addr":"172.21.15.67:6828/4049884108","heartbeat_front_addr":"172.21.15.67:6825/4049884108","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":787,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":699,"last_clean_end":738,"up_from":747,"up_thru":816,"down_at":739,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6801","nonce":3161027348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6803","nonce":3161027348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6807","nonce":3161027348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6805","nonce":3161027348}]},"public_addr":"172.21.15.67:6801/3161027348","cluster_addr":"172.21.15.67:6803/3161027348","heartbeat_back_addr":"172.21.15.67:6807/3161027348","heartbeat_front_addr":"172.21.15.67:6805/3161027348","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":677,"last_clean_end":785,"up_from":794,"up_thru":811,"down_at":786,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":167623373},{"type":"v1","addr":"172.21.15.82:6803","nonce":167623373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":167623373},{"type":"v1","addr":"172.21.15.82:6807","nonce":167623373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":167623373},{"type":"v1","addr":"172.21.15.82:6815","nonce":167623373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":167623373},{"type":"v1","addr":"172.21.15.82:6811","nonce":167623373}]},"public_addr":"172.21.15.82:6803/167623373","cluster_addr":"172.21.15.82:6807/167623373","heartbeat_back_addr":"172.21.15.82:6815/167623373","heartbeat_front_addr":"172.21.15.82:6811/167623373","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":515,"last_clean_end":810,"up_from":816,"up_thru":816,"down_at":811,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":1480534728},{"type":"v1","addr":"172.21.15.82:6817","nonce":1480534728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":1480534728},{"type":"v1","addr":"172.21.15.82:6819","nonce":1480534728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":1480534728},{"type":"v1","addr":"172.21.15.82:6823","nonce":1480534728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":1480534728},{"type":"v1","addr":"172.21.15.82:6821","nonce":1480534728}]},"public_addr":"172.21.15.82:6817/1480534728","cluster_addr":"172.21.15.82:6819/1480534728","heartbeat_back_addr":"172.21.15.82:6823/1480534728","heartbeat_front_addr":"172.21.15.82:6821/1480534728","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":629,"last_clean_end":715,"up_from":722,"up_thru":816,"down_at":716,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6801","nonce":326680821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6806","nonce":326680821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6814","nonce":326680821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6810","nonce":326680821}]},"public_addr":"172.21.15.82:6801/326680821","cluster_addr":"172.21.15.82:6806/326680821","heartbeat_back_addr":"172.21.15.82:6814/326680821","heartbeat_front_addr":"172.21.15.82:6810/326680821","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":604,"last_clean_end":645,"up_from":650,"up_thru":794,"down_at":646,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6825","nonce":1497758647}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6827","nonce":1497758647}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6831","nonce":1497758647}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":1497758647},{"type":"v1","addr":"172.21.15.82:6829","nonce":1497758647}]},"public_addr":"172.21.15.82:6825/1497758647","cluster_addr":"172.21.15.82:6827/1497758647","heartbeat_back_addr":"172.21.15.82:6831/1497758647","heartbeat_front_addr":"172.21.15.82:6829/1497758647","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:25:34.584003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:25:11.129246+0000","dead_epoch":764},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:24:57.221244+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:25:27.862088+0000","dead_epoch":739},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:24:33.848220+0000","dead_epoch":786},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:25:24.831493+0000","dead_epoch":811},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:24:46.784109+0000","dead_epoch":716},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:25:21.205496+0000","dead_epoch":646}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:25:36.705 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:25:36.705 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:25:36.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:36.823+0000 7fdb7f0ed640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:36.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:36.843+0000 7fdb7f0ed640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:37.016 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:25:37.034 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:25:37.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:37.143+0000 7fbccd597640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:37.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:37.159+0000 7fbccd597640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:37.338 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:25:37.339 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:25:37.358 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:25:37.358 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:25:37.358 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:25:37.358 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:25:37.358 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 7, 6, 3, 1, 4, 5] 2024-03-30T13:25:37.358 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:25:37.358 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:25:37.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:37.467+0000 7f82eb71d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:37.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:37.479+0000 7f82eb71d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:37.648 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:25:37.649 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:25:37.664 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1727,"stamp":"2024-03-30T13:25:36.307107+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":313412745,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":459328,"kb_used_data":149964,"kb_used_omap":73,"kb_used_meta":309174,"kb_avail":754515392,"statfs":{"total":773094113280,"available":772623761408,"internally_reserved":0,"allocated":153563136,"data_stored":122510806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":75753,"internal_metadata":316594199},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001202"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1868,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-30T13:25:34.999668+0000","last_change":"2024-03-30T13:25:20.550328+0000","last_active":"2024-03-30T13:25:34.999668+0000","last_peered":"2024-03-30T13:25:34.999668+0000","last_clean":"2024-03-30T13:25:34.999668+0000","last_became_active":"2024-03-30T13:25:20.549084+0000","last_became_peered":"2024-03-30T13:25:20.549084+0000","last_unstale":"2024-03-30T13:25:34.999668+0000","last_undegraded":"2024-03-30T13:25:34.999668+0000","last_fullsized":"2024-03-30T13:25:34.999668+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:25:12.543076+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:25:12.543076+0000","last_clean_scrub_stamp":"2024-03-30T13:25:12.543076+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:32.027663+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":13764735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1728,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-03-30T13:25:30.650008+0000","last_change":"2024-03-30T13:24:59.916106+0000","last_active":"2024-03-30T13:25:30.650008+0000","last_peered":"2024-03-30T13:25:30.650008+0000","last_clean":"2024-03-30T13:25:30.650008+0000","last_became_active":"2024-03-30T13:24:29.438667+0000","last_became_peered":"2024-03-30T13:24:29.438667+0000","last_unstale":"2024-03-30T13:25:30.650008+0000","last_undegraded":"2024-03-30T13:25:30.650008+0000","last_fullsized":"2024-03-30T13:25:30.650008+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:24:59.915857+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:24:59.915857+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:12.958686+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":554,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-03-30T13:25:30.294336+0000","last_change":"2024-03-30T13:24:29.442221+0000","last_active":"2024-03-30T13:25:30.294336+0000","last_peered":"2024-03-30T13:25:30.294336+0000","last_clean":"2024-03-30T13:25:30.294336+0000","last_became_active":"2024-03-30T13:24:29.440460+0000","last_became_peered":"2024-03-30T13:24:29.440460+0000","last_unstale":"2024-03-30T13:25:30.294336+0000","last_undegraded":"2024-03-30T13:25:30.294336+0000","last_fullsized":"2024-03-30T13:25:30.294336+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:24:27.040925+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:24:27.040925+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:51.105436+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":39075780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":529,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-03-30T13:25:30.294021+0000","last_change":"2024-03-30T13:24:53.286146+0000","last_active":"2024-03-30T13:25:30.294021+0000","last_peered":"2024-03-30T13:25:30.294021+0000","last_clean":"2024-03-30T13:25:30.294021+0000","last_became_active":"2024-03-30T13:21:56.541153+0000","last_became_peered":"2024-03-30T13:21:56.541153+0000","last_unstale":"2024-03-30T13:25:30.294021+0000","last_undegraded":"2024-03-30T13:25:30.294021+0000","last_fullsized":"2024-03-30T13:25:30.294021+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:24:53.285743+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:24:53.285743+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:10.430742+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":668,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-03-30T13:25:34.211450+0000","last_change":"2024-03-30T13:25:20.557379+0000","last_active":"2024-03-30T13:25:34.211450+0000","last_peered":"2024-03-30T13:25:34.211450+0000","last_clean":"2024-03-30T13:25:34.211450+0000","last_became_active":"2024-03-30T13:25:20.556165+0000","last_became_peered":"2024-03-30T13:25:20.556165+0000","last_unstale":"2024-03-30T13:25:34.211450+0000","last_undegraded":"2024-03-30T13:25:34.211450+0000","last_fullsized":"2024-03-30T13:25:34.211450+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:24:24.431636+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:24:24.431636+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:47.484058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1798,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-03-30T13:25:30.651324+0000","last_change":"2024-03-30T13:25:20.552405+0000","last_active":"2024-03-30T13:25:30.651324+0000","last_peered":"2024-03-30T13:25:30.651324+0000","last_clean":"2024-03-30T13:25:30.651324+0000","last_became_active":"2024-03-30T13:25:20.551941+0000","last_became_peered":"2024-03-30T13:25:20.551941+0000","last_unstale":"2024-03-30T13:25:30.651324+0000","last_undegraded":"2024-03-30T13:25:30.651324+0000","last_fullsized":"2024-03-30T13:25:30.651324+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:25:01.341334+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:25:01.341334+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:21.421274+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11394981,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":588,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-03-30T13:25:30.294080+0000","last_change":"2024-03-30T13:24:32.092922+0000","last_active":"2024-03-30T13:25:30.294080+0000","last_peered":"2024-03-30T13:25:30.294080+0000","last_clean":"2024-03-30T13:25:30.294080+0000","last_became_active":"2024-03-30T13:24:29.440149+0000","last_became_peered":"2024-03-30T13:24:29.440149+0000","last_unstale":"2024-03-30T13:25:30.294080+0000","last_undegraded":"2024-03-30T13:25:30.294080+0000","last_fullsized":"2024-03-30T13:25:30.294080+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:24:32.092620+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:24:32.092620+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:42.584351+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":43916097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1155,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-03-30T13:25:31.994865+0000","last_change":"2024-03-30T13:25:20.551362+0000","last_active":"2024-03-30T13:25:31.994865+0000","last_peered":"2024-03-30T13:25:31.994865+0000","last_clean":"2024-03-30T13:25:31.994865+0000","last_became_active":"2024-03-30T13:25:20.549997+0000","last_became_peered":"2024-03-30T13:25:20.549997+0000","last_unstale":"2024-03-30T13:25:31.994865+0000","last_undegraded":"2024-03-30T13:25:31.994865+0000","last_fullsized":"2024-03-30T13:25:31.994865+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:24:35.540394+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:24:35.540394+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:48.570974+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1895,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-03-30T13:25:30.651289+0000","last_change":"2024-03-30T13:25:20.553343+0000","last_active":"2024-03-30T13:25:30.651289+0000","last_peered":"2024-03-30T13:25:30.651289+0000","last_clean":"2024-03-30T13:25:30.651289+0000","last_became_active":"2024-03-30T13:25:20.552377+0000","last_became_peered":"2024-03-30T13:25:20.552377+0000","last_unstale":"2024-03-30T13:25:30.651289+0000","last_undegraded":"2024-03-30T13:25:30.651289+0000","last_fullsized":"2024-03-30T13:25:30.651289+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:24:46.148128+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:24:46.148128+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:10.093152+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":65983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1550,"reported_epoch":827,"state":"active+clean","last_fresh":"2024-03-30T13:25:30.649863+0000","last_change":"2024-03-30T13:24:30.839444+0000","last_active":"2024-03-30T13:25:30.649863+0000","last_peered":"2024-03-30T13:25:30.649863+0000","last_clean":"2024-03-30T13:25:30.649863+0000","last_became_active":"2024-03-30T13:24:29.442802+0000","last_became_peered":"2024-03-30T13:24:29.442802+0000","last_unstale":"2024-03-30T13:25:30.649863+0000","last_undegraded":"2024-03-30T13:25:30.649863+0000","last_fullsized":"2024-03-30T13:25:30.649863+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:24:30.839164+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:24:30.839164+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:38.428401+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1875,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-30T13:25:34.999565+0000","last_change":"2024-03-30T13:24:43.915621+0000","last_active":"2024-03-30T13:25:34.999565+0000","last_peered":"2024-03-30T13:25:34.999565+0000","last_clean":"2024-03-30T13:25:34.999565+0000","last_became_active":"2024-03-30T13:24:29.447202+0000","last_became_peered":"2024-03-30T13:24:29.447202+0000","last_unstale":"2024-03-30T13:25:34.999565+0000","last_undegraded":"2024-03-30T13:25:34.999565+0000","last_fullsized":"2024-03-30T13:25:34.999565+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:24:43.915348+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:24:43.915348+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:58.654177+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1969,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-30T13:25:34.999536+0000","last_change":"2024-03-30T13:25:20.551031+0000","last_active":"2024-03-30T13:25:34.999536+0000","last_peered":"2024-03-30T13:25:34.999536+0000","last_clean":"2024-03-30T13:25:34.999536+0000","last_became_active":"2024-03-30T13:25:20.549465+0000","last_became_peered":"2024-03-30T13:25:20.549465+0000","last_unstale":"2024-03-30T13:25:34.999536+0000","last_undegraded":"2024-03-30T13:25:34.999536+0000","last_fullsized":"2024-03-30T13:25:34.999536+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:24:56.750718+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:24:56.750718+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:09.232296+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":721,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-03-30T13:25:34.211642+0000","last_change":"2024-03-30T13:25:20.551619+0000","last_active":"2024-03-30T13:25:34.211642+0000","last_peered":"2024-03-30T13:25:34.211642+0000","last_clean":"2024-03-30T13:25:34.211642+0000","last_became_active":"2024-03-30T13:25:20.549427+0000","last_became_peered":"2024-03-30T13:25:20.549427+0000","last_unstale":"2024-03-30T13:25:34.211642+0000","last_undegraded":"2024-03-30T13:25:34.211642+0000","last_fullsized":"2024-03-30T13:25:34.211642+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:25:15.484555+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:25:15.484555+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:34.070118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":532,"reported_epoch":826,"state":"active+clean","last_fresh":"2024-03-30T13:25:30.294420+0000","last_change":"2024-03-30T13:25:12.391849+0000","last_active":"2024-03-30T13:25:30.294420+0000","last_peered":"2024-03-30T13:25:30.294420+0000","last_clean":"2024-03-30T13:25:30.294420+0000","last_became_active":"2024-03-30T13:22:47.489008+0000","last_became_peered":"2024-03-30T13:22:47.489008+0000","last_unstale":"2024-03-30T13:25:30.294420+0000","last_undegraded":"2024-03-30T13:25:30.294420+0000","last_fullsized":"2024-03-30T13:25:30.294420+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:25:12.391479+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:25:12.391479+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:26.494439+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4974806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":736,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-03-30T13:25:34.211382+0000","last_change":"2024-03-30T13:25:20.558649+0000","last_active":"2024-03-30T13:25:34.211382+0000","last_peered":"2024-03-30T13:25:34.211382+0000","last_clean":"2024-03-30T13:25:34.211382+0000","last_became_active":"2024-03-30T13:25:20.556235+0000","last_became_peered":"2024-03-30T13:25:20.556235+0000","last_unstale":"2024-03-30T13:25:34.211382+0000","last_undegraded":"2024-03-30T13:25:34.211382+0000","last_fullsized":"2024-03-30T13:25:34.211382+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_clean_scrub_stamp":"2024-03-30T13:24:40.633617+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:08.997018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":1016,"reported_epoch":828,"state":"active+clean","last_fresh":"2024-03-30T13:25:31.994960+0000","last_change":"2024-03-30T13:25:20.552502+0000","last_active":"2024-03-30T13:25:31.994960+0000","last_peered":"2024-03-30T13:25:31.994960+0000","last_clean":"2024-03-30T13:25:31.994960+0000","last_became_active":"2024-03-30T13:25:20.550231+0000","last_became_peered":"2024-03-30T13:25:20.550231+0000","last_unstale":"2024-03-30T13:25:31.994960+0000","last_undegraded":"2024-03-30T13:25:31.994960+0000","last_fullsized":"2024-03-30T13:25:31.994960+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:24:36.495187+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:24:36.495187+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:58.381669+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5226911,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":313412745,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387758,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59260,"kb_used_data":22064,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94312580,"statfs":{"total":96636764160,"available":96576081920,"internally_reserved":0,"allocated":22593536,"data_stored":18700093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":816,"seq":3504693313541,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68608,"kb_used_data":26796,"kb_used_omap":12,"kb_used_meta":41779,"kb_avail":94303232,"statfs":{"total":96636764160,"available":96566509568,"internally_reserved":0,"allocated":27439104,"data_stored":23552462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12654,"internal_metadata":42782354},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":794,"seq":3410204033039,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59000,"kb_used_data":18476,"kb_used_omap":15,"kb_used_meta":40496,"kb_avail":94312840,"statfs":{"total":96636764160,"available":96576348160,"internally_reserved":0,"allocated":18919424,"data_stored":15046384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15941,"internal_metadata":41468347},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509804,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52784,"kb_used_data":13280,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94319056,"statfs":{"total":96636764160,"available":96582713344,"internally_reserved":0,"allocated":13598720,"data_stored":9705277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":769,"seq":3302829850649,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50176,"kb_used_data":12532,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94321664,"statfs":{"total":96636764160,"available":96585383936,"internally_reserved":0,"allocated":12832768,"data_stored":8952927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10995,"internal_metadata":38524173},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509805,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55796,"kb_used_data":20456,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94316044,"statfs":{"total":96636764160,"available":96579629056,"internally_reserved":0,"allocated":20946944,"data_stored":17053501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":650,"seq":2791728742477,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57548,"kb_used_data":16952,"kb_used_omap":15,"kb_used_meta":40560,"kb_avail":94314292,"statfs":{"total":96636764160,"available":96577835008,"internally_reserved":0,"allocated":17358848,"data_stored":13492609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15933,"internal_metadata":41533891},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570147,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56156,"kb_used_data":19408,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94315684,"statfs":{"total":96636764160,"available":96579260416,"internally_reserved":0,"allocated":19873792,"data_stored":16007553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:25:37.666 INFO:tasks.thrashosds.thrasher:Choosing PG 1.0 with acting set [7, 0, 6] 2024-03-30T13:25:37.666 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 2, 7, 6, 3, 1, 4, 5] 2024-03-30T13:25:37.666 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-30T13:25:37.666 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:25:37.734 INFO:tasks.ceph.osd.7:Stopped 2024-03-30T13:25:37.734 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 2, 7, 6, 3, 1, 4, 5] 2024-03-30T13:25:37.734 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 7 2024-03-30T13:25:37.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:37.843+0000 7f506c953640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:37.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:37.859+0000 7f506c953640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:38.042 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:25:38.039+0000 7fbb38f08640 -1 mon.a@0(leader).osd e832 definitely_dead 0 2024-03-30T13:25:38.744 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.7. 2024-03-30T13:25:38.772 INFO:tasks.thrashosds.thrasher:dead_osds=[7], live_osds=[0, 2, 6, 3, 1, 4, 5] 2024-03-30T13:25:38.772 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:25:38.772 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:25:38.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:38.883+0000 7fc2a0bfe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:38.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:25:38.895+0000 7fc2a0bfe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:25:39.068 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:25:39.068 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:25:39.085 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1729,"stamp":"2024-03-30T13:25:38.742266+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":313412745,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":42,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":401960,"kb_used_data":133180,"kb_used_omap":58,"kb_used_meta":268613,"kb_avail":660200920,"statfs":{"total":676457349120,"available":676045742080,"internally_reserved":0,"allocated":136376320,"data_stored":109095165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59820,"internal_metadata":275060308},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.051508"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1868,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-30T13:25:34.999668+0000","last_change":"2024-03-30T13:25:20.550328+0000","last_active":"2024-03-30T13:25:34.999668+0000","last_peered":"2024-03-30T13:25:34.999668+0000","last_clean":"2024-03-30T13:25:34.999668+0000","last_became_active":"2024-03-30T13:25:20.549084+0000","last_became_peered":"2024-03-30T13:25:20.549084+0000","last_unstale":"2024-03-30T13:25:34.999668+0000","last_undegraded":"2024-03-30T13:25:34.999668+0000","last_fullsized":"2024-03-30T13:25:34.999668+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:25:12.543076+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:25:12.543076+0000","last_clean_scrub_stamp":"2024-03-30T13:25:12.543076+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:32.027663+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":13764735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1737,"reported_epoch":832,"state":"active+clean","last_fresh":"2024-03-30T13:25:36.098627+0000","last_change":"2024-03-30T13:24:59.916106+0000","last_active":"2024-03-30T13:25:36.098627+0000","last_peered":"2024-03-30T13:25:36.098627+0000","last_clean":"2024-03-30T13:25:36.098627+0000","last_became_active":"2024-03-30T13:24:29.438667+0000","last_became_peered":"2024-03-30T13:24:29.438667+0000","last_unstale":"2024-03-30T13:25:36.098627+0000","last_undegraded":"2024-03-30T13:25:36.098627+0000","last_fullsized":"2024-03-30T13:25:36.098627+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:24:59.915857+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:24:59.915857+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:12.958686+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2655473,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":565,"reported_epoch":832,"state":"stale+active+clean","last_fresh":"2024-03-30T13:25:35.768142+0000","last_change":"2024-03-30T13:24:29.442221+0000","last_active":"2024-03-30T13:25:35.768142+0000","last_peered":"2024-03-30T13:25:35.768142+0000","last_clean":"2024-03-30T13:25:35.768142+0000","last_became_active":"2024-03-30T13:24:29.440460+0000","last_became_peered":"2024-03-30T13:24:29.440460+0000","last_unstale":"2024-03-30T13:25:38.742286+0000","last_undegraded":"2024-03-30T13:25:35.768142+0000","last_fullsized":"2024-03-30T13:25:35.768142+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:24:27.040925+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:24:27.040925+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:51.105436+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":39075780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":540,"reported_epoch":832,"state":"stale+active+clean","last_fresh":"2024-03-30T13:25:35.768044+0000","last_change":"2024-03-30T13:24:53.286146+0000","last_active":"2024-03-30T13:25:35.768044+0000","last_peered":"2024-03-30T13:25:35.768044+0000","last_clean":"2024-03-30T13:25:35.768044+0000","last_became_active":"2024-03-30T13:21:56.541153+0000","last_became_peered":"2024-03-30T13:21:56.541153+0000","last_unstale":"2024-03-30T13:25:38.742288+0000","last_undegraded":"2024-03-30T13:25:35.768044+0000","last_fullsized":"2024-03-30T13:25:35.768044+0000","mapping_epoch":722,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":723,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:24:53.285743+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:24:53.285743+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:10.430742+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":668,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-03-30T13:25:34.211450+0000","last_change":"2024-03-30T13:25:20.557379+0000","last_active":"2024-03-30T13:25:34.211450+0000","last_peered":"2024-03-30T13:25:34.211450+0000","last_clean":"2024-03-30T13:25:34.211450+0000","last_became_active":"2024-03-30T13:25:20.556165+0000","last_became_peered":"2024-03-30T13:25:20.556165+0000","last_unstale":"2024-03-30T13:25:34.211450+0000","last_undegraded":"2024-03-30T13:25:34.211450+0000","last_fullsized":"2024-03-30T13:25:34.211450+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:24:24.431636+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:24:24.431636+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:47.484058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1807,"reported_epoch":832,"state":"active+clean","last_fresh":"2024-03-30T13:25:35.706022+0000","last_change":"2024-03-30T13:25:20.552405+0000","last_active":"2024-03-30T13:25:35.706022+0000","last_peered":"2024-03-30T13:25:35.706022+0000","last_clean":"2024-03-30T13:25:35.706022+0000","last_became_active":"2024-03-30T13:25:20.551941+0000","last_became_peered":"2024-03-30T13:25:20.551941+0000","last_unstale":"2024-03-30T13:25:35.706022+0000","last_undegraded":"2024-03-30T13:25:35.706022+0000","last_fullsized":"2024-03-30T13:25:35.706022+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:25:01.341334+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:25:01.341334+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:21.421274+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11394981,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":600,"reported_epoch":832,"state":"stale+active+clean","last_fresh":"2024-03-30T13:25:35.768255+0000","last_change":"2024-03-30T13:24:32.092922+0000","last_active":"2024-03-30T13:25:35.768255+0000","last_peered":"2024-03-30T13:25:35.768255+0000","last_clean":"2024-03-30T13:25:35.768255+0000","last_became_active":"2024-03-30T13:24:29.440149+0000","last_became_peered":"2024-03-30T13:24:29.440149+0000","last_unstale":"2024-03-30T13:25:38.742291+0000","last_undegraded":"2024-03-30T13:25:35.768255+0000","last_fullsized":"2024-03-30T13:25:35.768255+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:24:32.092620+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:24:32.092620+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:42.584351+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":43916097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1163,"reported_epoch":832,"state":"active+clean","last_fresh":"2024-03-30T13:25:36.098084+0000","last_change":"2024-03-30T13:25:20.551362+0000","last_active":"2024-03-30T13:25:36.098084+0000","last_peered":"2024-03-30T13:25:36.098084+0000","last_clean":"2024-03-30T13:25:36.098084+0000","last_became_active":"2024-03-30T13:25:20.549997+0000","last_became_peered":"2024-03-30T13:25:20.549997+0000","last_unstale":"2024-03-30T13:25:36.098084+0000","last_undegraded":"2024-03-30T13:25:36.098084+0000","last_fullsized":"2024-03-30T13:25:36.098084+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:24:35.540394+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:24:35.540394+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:48.570974+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1904,"reported_epoch":832,"state":"active+clean","last_fresh":"2024-03-30T13:25:35.706030+0000","last_change":"2024-03-30T13:25:20.553343+0000","last_active":"2024-03-30T13:25:35.706030+0000","last_peered":"2024-03-30T13:25:35.706030+0000","last_clean":"2024-03-30T13:25:35.706030+0000","last_became_active":"2024-03-30T13:25:20.552377+0000","last_became_peered":"2024-03-30T13:25:20.552377+0000","last_unstale":"2024-03-30T13:25:35.706030+0000","last_undegraded":"2024-03-30T13:25:35.706030+0000","last_fullsized":"2024-03-30T13:25:35.706030+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:24:46.148128+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:24:46.148128+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:10.093152+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":65983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1559,"reported_epoch":832,"state":"active+clean","last_fresh":"2024-03-30T13:25:36.098381+0000","last_change":"2024-03-30T13:24:30.839444+0000","last_active":"2024-03-30T13:25:36.098381+0000","last_peered":"2024-03-30T13:25:36.098381+0000","last_clean":"2024-03-30T13:25:36.098381+0000","last_became_active":"2024-03-30T13:24:29.442802+0000","last_became_peered":"2024-03-30T13:24:29.442802+0000","last_unstale":"2024-03-30T13:25:36.098381+0000","last_undegraded":"2024-03-30T13:25:36.098381+0000","last_fullsized":"2024-03-30T13:25:36.098381+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:24:30.839164+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:24:30.839164+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:38.428401+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1875,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-30T13:25:34.999565+0000","last_change":"2024-03-30T13:24:43.915621+0000","last_active":"2024-03-30T13:25:34.999565+0000","last_peered":"2024-03-30T13:25:34.999565+0000","last_clean":"2024-03-30T13:25:34.999565+0000","last_became_active":"2024-03-30T13:24:29.447202+0000","last_became_peered":"2024-03-30T13:24:29.447202+0000","last_unstale":"2024-03-30T13:25:34.999565+0000","last_undegraded":"2024-03-30T13:25:34.999565+0000","last_fullsized":"2024-03-30T13:25:34.999565+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:24:43.915348+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:24:43.915348+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:58.654177+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1969,"reported_epoch":831,"state":"active+clean","last_fresh":"2024-03-30T13:25:34.999536+0000","last_change":"2024-03-30T13:25:20.551031+0000","last_active":"2024-03-30T13:25:34.999536+0000","last_peered":"2024-03-30T13:25:34.999536+0000","last_clean":"2024-03-30T13:25:34.999536+0000","last_became_active":"2024-03-30T13:25:20.549465+0000","last_became_peered":"2024-03-30T13:25:20.549465+0000","last_unstale":"2024-03-30T13:25:34.999536+0000","last_undegraded":"2024-03-30T13:25:34.999536+0000","last_fullsized":"2024-03-30T13:25:34.999536+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:24:56.750718+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:24:56.750718+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:09.232296+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":721,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-03-30T13:25:34.211642+0000","last_change":"2024-03-30T13:25:20.551619+0000","last_active":"2024-03-30T13:25:34.211642+0000","last_peered":"2024-03-30T13:25:34.211642+0000","last_clean":"2024-03-30T13:25:34.211642+0000","last_became_active":"2024-03-30T13:25:20.549427+0000","last_became_peered":"2024-03-30T13:25:20.549427+0000","last_unstale":"2024-03-30T13:25:34.211642+0000","last_undegraded":"2024-03-30T13:25:34.211642+0000","last_fullsized":"2024-03-30T13:25:34.211642+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:25:15.484555+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:25:15.484555+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:34.070118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":543,"reported_epoch":832,"state":"stale+active+clean","last_fresh":"2024-03-30T13:25:35.768067+0000","last_change":"2024-03-30T13:25:12.391849+0000","last_active":"2024-03-30T13:25:35.768067+0000","last_peered":"2024-03-30T13:25:35.768067+0000","last_clean":"2024-03-30T13:25:35.768067+0000","last_became_active":"2024-03-30T13:22:47.489008+0000","last_became_peered":"2024-03-30T13:22:47.489008+0000","last_unstale":"2024-03-30T13:25:38.742294+0000","last_undegraded":"2024-03-30T13:25:35.768067+0000","last_fullsized":"2024-03-30T13:25:35.768067+0000","mapping_epoch":747,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":748,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:25:12.391479+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:25:12.391479+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:26.494439+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4974806,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":736,"reported_epoch":830,"state":"active+clean","last_fresh":"2024-03-30T13:25:34.211382+0000","last_change":"2024-03-30T13:25:20.558649+0000","last_active":"2024-03-30T13:25:34.211382+0000","last_peered":"2024-03-30T13:25:34.211382+0000","last_clean":"2024-03-30T13:25:34.211382+0000","last_became_active":"2024-03-30T13:25:20.556235+0000","last_became_peered":"2024-03-30T13:25:20.556235+0000","last_unstale":"2024-03-30T13:25:34.211382+0000","last_undegraded":"2024-03-30T13:25:34.211382+0000","last_fullsized":"2024-03-30T13:25:34.211382+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_clean_scrub_stamp":"2024-03-30T13:24:40.633617+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:08.997018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":1024,"reported_epoch":832,"state":"active+clean","last_fresh":"2024-03-30T13:25:36.098106+0000","last_change":"2024-03-30T13:25:20.552502+0000","last_active":"2024-03-30T13:25:36.098106+0000","last_peered":"2024-03-30T13:25:36.098106+0000","last_clean":"2024-03-30T13:25:36.098106+0000","last_became_active":"2024-03-30T13:25:20.550231+0000","last_became_peered":"2024-03-30T13:25:20.550231+0000","last_unstale":"2024-03-30T13:25:36.098106+0000","last_undegraded":"2024-03-30T13:25:36.098106+0000","last_fullsized":"2024-03-30T13:25:36.098106+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:24:36.495187+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:24:36.495187+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:58.381669+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5226911,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":313412745,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387758,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59260,"kb_used_data":22064,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94312580,"statfs":{"total":96636764160,"available":96576081920,"internally_reserved":0,"allocated":22593536,"data_stored":18700093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":816,"seq":3504693313541,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68608,"kb_used_data":26796,"kb_used_omap":12,"kb_used_meta":41779,"kb_avail":94303232,"statfs":{"total":96636764160,"available":96566509568,"internally_reserved":0,"allocated":27439104,"data_stored":23552462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12654,"internal_metadata":42782354},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":794,"seq":3410204033040,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59060,"kb_used_data":18536,"kb_used_omap":15,"kb_used_meta":40496,"kb_avail":94312780,"statfs":{"total":96636764160,"available":96576286720,"internally_reserved":0,"allocated":18980864,"data_stored":15073869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15941,"internal_metadata":41468347},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509804,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52784,"kb_used_data":13280,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94319056,"statfs":{"total":96636764160,"available":96582713344,"internally_reserved":0,"allocated":13598720,"data_stored":9705277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":769,"seq":3302829850650,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50232,"kb_used_data":12580,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94321608,"statfs":{"total":96636764160,"available":96585326592,"internally_reserved":0,"allocated":12881920,"data_stored":8974925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10995,"internal_metadata":38524173},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509805,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55796,"kb_used_data":20456,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94316044,"statfs":{"total":96636764160,"available":96579629056,"internally_reserved":0,"allocated":20946944,"data_stored":17053501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570148,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56220,"kb_used_data":19468,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94315620,"statfs":{"total":96636764160,"available":96579194880,"internally_reserved":0,"allocated":19935232,"data_stored":16035038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:25:39.086 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:26:04.087 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:26:04.088 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:26:04.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:04.214+0000 7f9ed6778640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:04.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:04.226+0000 7f9ed6778640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:04.402 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:26:04.403 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:26:04.421 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1744,"stamp":"2024-03-30T13:26:04.316449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":273748753,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":412340,"kb_used_data":143596,"kb_used_omap":58,"kb_used_meta":268613,"kb_avail":660190540,"statfs":{"total":676457349120,"available":676035112960,"internally_reserved":0,"allocated":147042304,"data_stored":119552324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":59820,"internal_metadata":275060308},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.873686"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1876,"reported_epoch":835,"state":"active+clean","last_fresh":"2024-03-30T13:25:53.715378+0000","last_change":"2024-03-30T13:25:20.550328+0000","last_active":"2024-03-30T13:25:53.715378+0000","last_peered":"2024-03-30T13:25:53.715378+0000","last_clean":"2024-03-30T13:25:53.715378+0000","last_became_active":"2024-03-30T13:25:20.549084+0000","last_became_peered":"2024-03-30T13:25:20.549084+0000","last_unstale":"2024-03-30T13:25:53.715378+0000","last_undegraded":"2024-03-30T13:25:53.715378+0000","last_fullsized":"2024-03-30T13:25:53.715378+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:25:12.543076+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:25:12.543076+0000","last_clean_scrub_stamp":"2024-03-30T13:25:12.543076+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:32.027663+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":13764735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1753,"reported_epoch":835,"state":"active+clean","last_fresh":"2024-03-30T13:25:53.445239+0000","last_change":"2024-03-30T13:25:39.790324+0000","last_active":"2024-03-30T13:25:53.445239+0000","last_peered":"2024-03-30T13:25:53.445239+0000","last_clean":"2024-03-30T13:25:53.445239+0000","last_became_active":"2024-03-30T13:25:39.759383+0000","last_became_peered":"2024-03-30T13:25:39.759383+0000","last_unstale":"2024-03-30T13:25:53.445239+0000","last_undegraded":"2024-03-30T13:25:53.445239+0000","last_fullsized":"2024-03-30T13:25:53.445239+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:24:59.915857+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:24:59.915857+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:20.227499+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5310946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":29,"reported_epoch":835,"state":"active+clean","last_fresh":"2024-03-30T13:25:53.716523+0000","last_change":"2024-03-30T13:25:41.166083+0000","last_active":"2024-03-30T13:25:53.716523+0000","last_peered":"2024-03-30T13:25:53.716523+0000","last_clean":"2024-03-30T13:25:53.716523+0000","last_became_active":"2024-03-30T13:25:39.757082+0000","last_became_peered":"2024-03-30T13:25:39.757082+0000","last_unstale":"2024-03-30T13:25:53.716523+0000","last_undegraded":"2024-03-30T13:25:53.716523+0000","last_fullsized":"2024-03-30T13:25:53.716523+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:25:41.165836+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:25:41.165836+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:01.780909+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":19,"reported_epoch":835,"state":"active+clean","last_fresh":"2024-03-30T13:25:53.447766+0000","last_change":"2024-03-30T13:25:39.818805+0000","last_active":"2024-03-30T13:25:53.447766+0000","last_peered":"2024-03-30T13:25:53.447766+0000","last_clean":"2024-03-30T13:25:53.447766+0000","last_became_active":"2024-03-30T13:25:39.761734+0000","last_became_peered":"2024-03-30T13:25:39.761734+0000","last_unstale":"2024-03-30T13:25:53.447766+0000","last_undegraded":"2024-03-30T13:25:53.447766+0000","last_fullsized":"2024-03-30T13:25:53.447766+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:24:53.285743+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:24:53.285743+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:20.364358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":686,"reported_epoch":835,"state":"active+clean","last_fresh":"2024-03-30T13:25:53.714748+0000","last_change":"2024-03-30T13:25:50.791314+0000","last_active":"2024-03-30T13:25:53.714748+0000","last_peered":"2024-03-30T13:25:53.714748+0000","last_clean":"2024-03-30T13:25:53.714748+0000","last_became_active":"2024-03-30T13:25:20.556165+0000","last_became_peered":"2024-03-30T13:25:20.556165+0000","last_unstale":"2024-03-30T13:25:53.714748+0000","last_undegraded":"2024-03-30T13:25:53.714748+0000","last_fullsized":"2024-03-30T13:25:53.714748+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:25:50.791021+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:25:50.791021+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:01.537702+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1813,"reported_epoch":835,"state":"active+clean","last_fresh":"2024-03-30T13:25:53.715823+0000","last_change":"2024-03-30T13:25:20.552405+0000","last_active":"2024-03-30T13:25:53.715823+0000","last_peered":"2024-03-30T13:25:53.715823+0000","last_clean":"2024-03-30T13:25:53.715823+0000","last_became_active":"2024-03-30T13:25:20.551941+0000","last_became_peered":"2024-03-30T13:25:20.551941+0000","last_unstale":"2024-03-30T13:25:53.715823+0000","last_undegraded":"2024-03-30T13:25:53.715823+0000","last_fullsized":"2024-03-30T13:25:53.715823+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:25:01.341334+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:25:01.341334+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:21.421274+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11394981,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":42,"reported_epoch":835,"state":"active+clean","last_fresh":"2024-03-30T13:25:56.629158+0000","last_change":"2024-03-30T13:25:56.629248+0000","last_active":"2024-03-30T13:25:56.629158+0000","last_peered":"2024-03-30T13:25:56.629158+0000","last_clean":"2024-03-30T13:25:56.629158+0000","last_became_active":"2024-03-30T13:25:39.756837+0000","last_became_peered":"2024-03-30T13:25:39.756837+0000","last_unstale":"2024-03-30T13:25:56.629158+0000","last_undegraded":"2024-03-30T13:25:56.629158+0000","last_fullsized":"2024-03-30T13:25:56.629158+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:25:56.628976+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:25:56.628976+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:09.713434+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1178,"reported_epoch":835,"state":"active+clean","last_fresh":"2024-03-30T13:25:53.447766+0000","last_change":"2024-03-30T13:25:52.600165+0000","last_active":"2024-03-30T13:25:53.447766+0000","last_peered":"2024-03-30T13:25:53.447766+0000","last_clean":"2024-03-30T13:25:53.447766+0000","last_became_active":"2024-03-30T13:25:20.549997+0000","last_became_peered":"2024-03-30T13:25:20.549997+0000","last_unstale":"2024-03-30T13:25:53.447766+0000","last_undegraded":"2024-03-30T13:25:53.447766+0000","last_fullsized":"2024-03-30T13:25:53.447766+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:25:52.599859+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:25:52.599859+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:01.619779+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1922,"reported_epoch":835,"state":"active+clean","last_fresh":"2024-03-30T13:25:53.715894+0000","last_change":"2024-03-30T13:25:40.102600+0000","last_active":"2024-03-30T13:25:53.715894+0000","last_peered":"2024-03-30T13:25:53.715894+0000","last_clean":"2024-03-30T13:25:53.715894+0000","last_became_active":"2024-03-30T13:25:39.760207+0000","last_became_peered":"2024-03-30T13:25:39.760207+0000","last_unstale":"2024-03-30T13:25:53.715894+0000","last_undegraded":"2024-03-30T13:25:53.715894+0000","last_fullsized":"2024-03-30T13:25:53.715894+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:24:46.148128+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:24:46.148128+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:14.808122+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":71059072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1573,"reported_epoch":835,"state":"active+clean","last_fresh":"2024-03-30T13:25:53.445160+0000","last_change":"2024-03-30T13:25:40.607096+0000","last_active":"2024-03-30T13:25:53.445160+0000","last_peered":"2024-03-30T13:25:53.445160+0000","last_clean":"2024-03-30T13:25:53.445160+0000","last_became_active":"2024-03-30T13:24:29.442802+0000","last_became_peered":"2024-03-30T13:24:29.442802+0000","last_unstale":"2024-03-30T13:25:53.445160+0000","last_undegraded":"2024-03-30T13:25:53.445160+0000","last_fullsized":"2024-03-30T13:25:53.445160+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:25:40.606726+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:25:40.606726+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:46.778568+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1883,"reported_epoch":835,"state":"active+clean","last_fresh":"2024-03-30T13:25:53.714588+0000","last_change":"2024-03-30T13:24:43.915621+0000","last_active":"2024-03-30T13:25:53.714588+0000","last_peered":"2024-03-30T13:25:53.714588+0000","last_clean":"2024-03-30T13:25:53.714588+0000","last_became_active":"2024-03-30T13:24:29.447202+0000","last_became_peered":"2024-03-30T13:24:29.447202+0000","last_unstale":"2024-03-30T13:25:53.714588+0000","last_undegraded":"2024-03-30T13:25:53.714588+0000","last_fullsized":"2024-03-30T13:25:53.714588+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:24:43.915348+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:24:43.915348+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:25:58.654177+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":1977,"reported_epoch":835,"state":"active+clean","last_fresh":"2024-03-30T13:25:53.715349+0000","last_change":"2024-03-30T13:25:20.551031+0000","last_active":"2024-03-30T13:25:53.715349+0000","last_peered":"2024-03-30T13:25:53.715349+0000","last_clean":"2024-03-30T13:25:53.715349+0000","last_became_active":"2024-03-30T13:25:20.549465+0000","last_became_peered":"2024-03-30T13:25:20.549465+0000","last_unstale":"2024-03-30T13:25:53.715349+0000","last_undegraded":"2024-03-30T13:25:53.715349+0000","last_fullsized":"2024-03-30T13:25:53.715349+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:24:56.750718+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:24:56.750718+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:09.232296+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":731,"reported_epoch":835,"state":"active+clean","last_fresh":"2024-03-30T13:25:53.714883+0000","last_change":"2024-03-30T13:25:20.551619+0000","last_active":"2024-03-30T13:25:53.714883+0000","last_peered":"2024-03-30T13:25:53.714883+0000","last_clean":"2024-03-30T13:25:53.714883+0000","last_became_active":"2024-03-30T13:25:20.549427+0000","last_became_peered":"2024-03-30T13:25:20.549427+0000","last_unstale":"2024-03-30T13:25:53.714883+0000","last_undegraded":"2024-03-30T13:25:53.714883+0000","last_fullsized":"2024-03-30T13:25:53.714883+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:25:15.484555+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:25:15.484555+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:34.070118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":17,"reported_epoch":835,"state":"active+clean","last_fresh":"2024-03-30T13:25:53.714724+0000","last_change":"2024-03-30T13:25:39.787530+0000","last_active":"2024-03-30T13:25:53.714724+0000","last_peered":"2024-03-30T13:25:53.714724+0000","last_clean":"2024-03-30T13:25:53.714724+0000","last_became_active":"2024-03-30T13:25:39.755148+0000","last_became_peered":"2024-03-30T13:25:39.755148+0000","last_unstale":"2024-03-30T13:25:53.714724+0000","last_undegraded":"2024-03-30T13:25:53.714724+0000","last_fullsized":"2024-03-30T13:25:53.714724+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:25:12.391479+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:25:12.391479+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:36.704662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":746,"reported_epoch":835,"state":"active+clean","last_fresh":"2024-03-30T13:25:53.714949+0000","last_change":"2024-03-30T13:25:20.558649+0000","last_active":"2024-03-30T13:25:53.714949+0000","last_peered":"2024-03-30T13:25:53.714949+0000","last_clean":"2024-03-30T13:25:53.714949+0000","last_became_active":"2024-03-30T13:25:20.556235+0000","last_became_peered":"2024-03-30T13:25:20.556235+0000","last_unstale":"2024-03-30T13:25:53.714949+0000","last_undegraded":"2024-03-30T13:25:53.714949+0000","last_fullsized":"2024-03-30T13:25:53.714949+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_clean_scrub_stamp":"2024-03-30T13:24:40.633617+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:08.997018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":1039,"reported_epoch":835,"state":"active+clean","last_fresh":"2024-03-30T13:25:58.642356+0000","last_change":"2024-03-30T13:25:58.642430+0000","last_active":"2024-03-30T13:25:58.642356+0000","last_peered":"2024-03-30T13:25:58.642356+0000","last_clean":"2024-03-30T13:25:58.642356+0000","last_became_active":"2024-03-30T13:25:20.550231+0000","last_became_peered":"2024-03-30T13:25:20.550231+0000","last_unstale":"2024-03-30T13:25:58.642356+0000","last_undegraded":"2024-03-30T13:25:58.642356+0000","last_fullsized":"2024-03-30T13:25:58.642356+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:25:58.642180+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:25:58.642180+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:14.896158+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5226911,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":273748753,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":107036672,"data_stored":107036672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387763,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59320,"kb_used_data":22124,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94312520,"statfs":{"total":96636764160,"available":96576020480,"internally_reserved":0,"allocated":22654976,"data_stored":18727836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":816,"seq":3504693313546,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69312,"kb_used_data":27508,"kb_used_omap":12,"kb_used_meta":41779,"kb_avail":94302528,"statfs":{"total":96636764160,"available":96565788672,"internally_reserved":0,"allocated":28168192,"data_stored":24241052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12654,"internal_metadata":42782354},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":794,"seq":3410204033045,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59092,"kb_used_data":18572,"kb_used_omap":15,"kb_used_meta":40496,"kb_avail":94312748,"statfs":{"total":96636764160,"available":96576253952,"internally_reserved":0,"allocated":19017728,"data_stored":15090588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15941,"internal_metadata":41468347},"hb_peers":[0,1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509809,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58936,"kb_used_data":19436,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94312904,"statfs":{"total":96636764160,"available":96576413696,"internally_reserved":0,"allocated":19902464,"data_stored":15975324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":769,"seq":3302829850655,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53564,"kb_used_data":15924,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94318276,"statfs":{"total":96636764160,"available":96581914624,"internally_reserved":0,"allocated":16306176,"data_stored":12379036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10995,"internal_metadata":38524173},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509810,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55852,"kb_used_data":20516,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315988,"statfs":{"total":96636764160,"available":96579571712,"internally_reserved":0,"allocated":21008384,"data_stored":17081244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570153,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56264,"kb_used_data":19516,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94315576,"statfs":{"total":96636764160,"available":96579149824,"internally_reserved":0,"allocated":19984384,"data_stored":16057244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":9121792,"data_stored":9121792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12718080,"data_stored":12718080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20983808,"data_stored":20983808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:26:04.422 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:26:04.422 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:26:04.422 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2024-03-30T13:26:04.422 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-30T13:26:04.423 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-30T13:26:04.426 INFO:tasks.ceph.osd.7:Started 2024-03-30T13:26:04.427 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-30T13:26:04.495 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:26:04.492+0000 7f03e72e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:04.502 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:26:04.500+0000 7f03e72e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:04.503 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:26:04.500+0000 7f03e72e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:04.545 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T13:26:04.545 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T13:26:04.545 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T13:26:04.545 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T13:26:04.545 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T13:26:04.545 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T13:26:04.545 INFO:teuthology.orchestra.run.smithi082.stderr:config diff 2024-03-30T13:26:04.545 INFO:teuthology.orchestra.run.smithi082.stderr:config diff get 2024-03-30T13:26:04.545 INFO:teuthology.orchestra.run.smithi082.stderr:config get 2024-03-30T13:26:04.545 INFO:teuthology.orchestra.run.smithi082.stderr:config help [] 2024-03-30T13:26:04.545 INFO:teuthology.orchestra.run.smithi082.stderr:config set ... 2024-03-30T13:26:04.545 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T13:26:04.547 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:26:04.547 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-03-30T13:26:05.035 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:26:05.032+0000 7f03e72e27c0 -1 Falling back to public interface 2024-03-30T13:26:08.844 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:26:08.840+0000 7f03e72e27c0 -1 osd.7 832 log_to_monitors true 2024-03-30T13:26:09.548 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-30T13:26:09.677 INFO:teuthology.orchestra.run.smithi082.stderr:no valid command found; 10 closest matches: 2024-03-30T13:26:09.677 INFO:teuthology.orchestra.run.smithi082.stderr:0 2024-03-30T13:26:09.677 INFO:teuthology.orchestra.run.smithi082.stderr:1 2024-03-30T13:26:09.677 INFO:teuthology.orchestra.run.smithi082.stderr:2 2024-03-30T13:26:09.677 INFO:teuthology.orchestra.run.smithi082.stderr:abort 2024-03-30T13:26:09.678 INFO:teuthology.orchestra.run.smithi082.stderr:assert 2024-03-30T13:26:09.678 INFO:teuthology.orchestra.run.smithi082.stderr:bluefs debug_inject_read_zeros 2024-03-30T13:26:09.678 INFO:teuthology.orchestra.run.smithi082.stderr:bluefs files list 2024-03-30T13:26:09.678 INFO:teuthology.orchestra.run.smithi082.stderr:bluefs stats 2024-03-30T13:26:09.678 INFO:teuthology.orchestra.run.smithi082.stderr:bluestore allocator dump block 2024-03-30T13:26:09.678 INFO:teuthology.orchestra.run.smithi082.stderr:bluestore allocator fragmentation block 2024-03-30T13:26:09.678 INFO:teuthology.orchestra.run.smithi082.stderr:admin_socket: invalid command 2024-03-30T13:26:09.680 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:26:09.680 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-03-30T13:26:10.549 INFO:tasks.ceph.osd.7.smithi082.stderr:2024-03-30T13:26:10.548+0000 7f03df0b7640 -1 osd.7 832 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:26:14.681 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-03-30T13:26:14.811 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-03-30T13:26:14.812 INFO:teuthology.orchestra.run.smithi082.stdout: "ops": [], 2024-03-30T13:26:14.812 INFO:teuthology.orchestra.run.smithi082.stdout: "num_ops": 0 2024-03-30T13:26:14.812 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-03-30T13:26:14.825 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:26:14.826 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-03-30T13:26:14.826 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 7 2024-03-30T13:26:14.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:14.954+0000 7fb2bccf3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:14.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:14.970+0000 7fb2bccf3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:15.148 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:26:15.150+0000 7fbb38f08640 -1 mon.a@0(leader).osd e837 definitely_dead 0 2024-03-30T13:26:15.552 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.7. 2024-03-30T13:26:15.579 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-03-30T13:26:15.579 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:26:15.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:15.698+0000 7f643c23e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:15.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:15.714+0000 7f643c23e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:16.562 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:26:16.595 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:26:16.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:16.722+0000 7f3fc4204640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:16.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:16.738+0000 7f3fc4204640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:17.571 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:26:17.601 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:26:17.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:17.726+0000 7facfdc61640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:17.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:17.742+0000 7facfdc61640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:18.575 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:26:18.600 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:26:18.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:18.722+0000 7fac18eee640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:18.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:18.738+0000 7fac18eee640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:19.594 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:26:19.621 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:26:19.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:19.738+0000 7f77aed76640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:19.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:19.758+0000 7f77aed76640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:20.614 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:26:20.636 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:26:20.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:20.754+0000 7fa6d9508640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:20.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:20.770+0000 7fa6d9508640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:21.607 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:26:21.635 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:26:21.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:21.758+0000 7f821cee8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:21.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:21.774+0000 7f821cee8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:22.625 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:26:22.648 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:26:22.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:22.770+0000 7f82b97f0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:22.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:22.786+0000 7f82b97f0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:23.634 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:26:23.660 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:26:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:23.782+0000 7f701e599640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:23.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:23.798+0000 7f701e599640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:24.643 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:26:24.669 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:26:24.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:24.790+0000 7fbea9dc8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:24.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:24.806+0000 7fbea9dc8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:25.647 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:26:25.673 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:26:25.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:25.790+0000 7f4a0bcca640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:25.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:25.806+0000 7f4a0bcca640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:26.664 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:26:26.690 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:26:26.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:26.810+0000 7fd6245ca640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:26.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:26.826+0000 7fd6245ca640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:27.666 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:26:27.693 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:26:27.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:27.810+0000 7fbfda569640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:27.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:27.826+0000 7fbfda569640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:28.696 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:26:28.723 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:26:28.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:28.846+0000 7f3de6872640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:28.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:28.862+0000 7f3de6872640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:29.698 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:26:29.722 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:26:29.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:29.838+0000 7f48e1d34640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:29.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:29.858+0000 7f48e1d34640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:30.700 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:26:30.729 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:26:30.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:30.854+0000 7f82a9d09640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:30.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:30.870+0000 7f82a9d09640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:31.718 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:26:31.774 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:26:31.774 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:26:31.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:31.898+0000 7f911b214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:31.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:31.910+0000 7f911b214640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:32.087 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:26:32.088 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:26:32.107 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1776,"stamp":"2024-03-30T13:26:31.714382+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":281479874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":461976,"kb_used_data":152160,"kb_used_omap":75,"kb_used_meta":309620,"kb_avail":754512744,"statfs":{"total":773094113280,"available":772621049856,"internally_reserved":0,"allocated":155811840,"data_stored":123531549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77408,"internal_metadata":317051296},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.044975"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1910,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-03-30T13:26:29.703107+0000","last_change":"2024-03-30T13:25:20.550328+0000","last_active":"2024-03-30T13:26:29.703107+0000","last_peered":"2024-03-30T13:26:29.703107+0000","last_clean":"2024-03-30T13:26:29.703107+0000","last_became_active":"2024-03-30T13:25:20.549084+0000","last_became_peered":"2024-03-30T13:25:20.549084+0000","last_unstale":"2024-03-30T13:26:29.703107+0000","last_undegraded":"2024-03-30T13:26:29.703107+0000","last_fullsized":"2024-03-30T13:26:29.703107+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:25:12.543076+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:25:12.543076+0000","last_clean_scrub_stamp":"2024-03-30T13:25:12.543076+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:32.027663+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":13764735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1800,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-30T13:26:27.671887+0000","last_change":"2024-03-30T13:26:19.721841+0000","last_active":"2024-03-30T13:26:27.671887+0000","last_peered":"2024-03-30T13:26:27.671887+0000","last_clean":"2024-03-30T13:26:27.671887+0000","last_became_active":"2024-03-30T13:26:16.583003+0000","last_became_peered":"2024-03-30T13:26:16.583003+0000","last_unstale":"2024-03-30T13:26:27.671887+0000","last_undegraded":"2024-03-30T13:26:27.671887+0000","last_fullsized":"2024-03-30T13:26:27.671887+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:26:19.721595+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:26:19.721595+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:24.825128+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":45,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.002547+0000","last_change":"2024-03-30T13:26:16.646953+0000","last_active":"2024-03-30T13:26:31.002547+0000","last_peered":"2024-03-30T13:26:31.002547+0000","last_clean":"2024-03-30T13:26:31.002547+0000","last_became_active":"2024-03-30T13:26:16.583921+0000","last_became_peered":"2024-03-30T13:26:16.583921+0000","last_unstale":"2024-03-30T13:26:31.002547+0000","last_undegraded":"2024-03-30T13:26:31.002547+0000","last_fullsized":"2024-03-30T13:26:31.002547+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:25:41.165836+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:25:41.165836+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:46.866235+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":51,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.002422+0000","last_change":"2024-03-30T13:26:21.852195+0000","last_active":"2024-03-30T13:26:31.002422+0000","last_peered":"2024-03-30T13:26:31.002422+0000","last_clean":"2024-03-30T13:26:31.002422+0000","last_became_active":"2024-03-30T13:26:16.577250+0000","last_became_peered":"2024-03-30T13:26:16.577250+0000","last_unstale":"2024-03-30T13:26:31.002422+0000","last_undegraded":"2024-03-30T13:26:31.002422+0000","last_fullsized":"2024-03-30T13:26:31.002422+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:26:21.851948+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:26:21.851948+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:41.578957+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":716,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-30T13:26:28.432783+0000","last_change":"2024-03-30T13:25:50.791314+0000","last_active":"2024-03-30T13:26:28.432783+0000","last_peered":"2024-03-30T13:26:28.432783+0000","last_clean":"2024-03-30T13:26:28.432783+0000","last_became_active":"2024-03-30T13:25:20.556165+0000","last_became_peered":"2024-03-30T13:25:20.556165+0000","last_unstale":"2024-03-30T13:26:28.432783+0000","last_undegraded":"2024-03-30T13:26:28.432783+0000","last_fullsized":"2024-03-30T13:26:28.432783+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:25:50.791021+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:25:50.791021+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:01.537702+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1855,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.004313+0000","last_change":"2024-03-30T13:26:25.441051+0000","last_active":"2024-03-30T13:26:31.004313+0000","last_peered":"2024-03-30T13:26:31.004313+0000","last_clean":"2024-03-30T13:26:31.004313+0000","last_became_active":"2024-03-30T13:25:20.551941+0000","last_became_peered":"2024-03-30T13:25:20.551941+0000","last_unstale":"2024-03-30T13:26:31.004313+0000","last_undegraded":"2024-03-30T13:26:31.004313+0000","last_fullsized":"2024-03-30T13:26:31.004313+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:26:25.440798+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:26:25.440798+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:36.431590+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11394981,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":58,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.002420+0000","last_change":"2024-03-30T13:26:17.131560+0000","last_active":"2024-03-30T13:26:31.002420+0000","last_peered":"2024-03-30T13:26:31.002420+0000","last_clean":"2024-03-30T13:26:31.002420+0000","last_became_active":"2024-03-30T13:26:16.583861+0000","last_became_peered":"2024-03-30T13:26:16.583861+0000","last_unstale":"2024-03-30T13:26:31.002420+0000","last_undegraded":"2024-03-30T13:26:31.002420+0000","last_fullsized":"2024-03-30T13:26:31.002420+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:25:56.628976+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:25:56.628976+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:12.425669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1205,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-03-30T13:26:26.670894+0000","last_change":"2024-03-30T13:25:52.600165+0000","last_active":"2024-03-30T13:26:26.670894+0000","last_peered":"2024-03-30T13:26:26.670894+0000","last_clean":"2024-03-30T13:26:26.670894+0000","last_became_active":"2024-03-30T13:25:20.549997+0000","last_became_peered":"2024-03-30T13:25:20.549997+0000","last_unstale":"2024-03-30T13:26:26.670894+0000","last_undegraded":"2024-03-30T13:26:26.670894+0000","last_fullsized":"2024-03-30T13:26:26.670894+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:25:52.599859+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:25:52.599859+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:01.619779+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1976,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.004093+0000","last_change":"2024-03-30T13:26:19.523394+0000","last_active":"2024-03-30T13:26:31.004093+0000","last_peered":"2024-03-30T13:26:31.004093+0000","last_clean":"2024-03-30T13:26:31.004093+0000","last_became_active":"2024-03-30T13:26:16.582448+0000","last_became_peered":"2024-03-30T13:26:16.582448+0000","last_unstale":"2024-03-30T13:26:31.004093+0000","last_undegraded":"2024-03-30T13:26:31.004093+0000","last_fullsized":"2024-03-30T13:26:31.004093+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:26:19.523081+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:26:19.523081+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:39.559039+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":76134720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1602,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-30T13:26:27.671850+0000","last_change":"2024-03-30T13:25:40.607096+0000","last_active":"2024-03-30T13:26:27.671850+0000","last_peered":"2024-03-30T13:26:27.671850+0000","last_clean":"2024-03-30T13:26:27.671850+0000","last_became_active":"2024-03-30T13:24:29.442802+0000","last_became_peered":"2024-03-30T13:24:29.442802+0000","last_unstale":"2024-03-30T13:26:27.671850+0000","last_undegraded":"2024-03-30T13:26:27.671850+0000","last_fullsized":"2024-03-30T13:26:27.671850+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:25:40.606726+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:25:40.606726+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:46.778568+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1925,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-03-30T13:26:29.702870+0000","last_change":"2024-03-30T13:26:01.555623+0000","last_active":"2024-03-30T13:26:29.702870+0000","last_peered":"2024-03-30T13:26:29.702870+0000","last_clean":"2024-03-30T13:26:29.702870+0000","last_became_active":"2024-03-30T13:24:29.447202+0000","last_became_peered":"2024-03-30T13:24:29.447202+0000","last_unstale":"2024-03-30T13:26:29.702870+0000","last_undegraded":"2024-03-30T13:26:29.702870+0000","last_fullsized":"2024-03-30T13:26:29.702870+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:26:01.555372+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:26:01.555372+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:16.483568+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":2019,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-03-30T13:26:29.702909+0000","last_change":"2024-03-30T13:26:09.357933+0000","last_active":"2024-03-30T13:26:29.702909+0000","last_peered":"2024-03-30T13:26:29.702909+0000","last_clean":"2024-03-30T13:26:29.702909+0000","last_became_active":"2024-03-30T13:25:20.549465+0000","last_became_peered":"2024-03-30T13:25:20.549465+0000","last_unstale":"2024-03-30T13:26:29.702909+0000","last_undegraded":"2024-03-30T13:26:29.702909+0000","last_fullsized":"2024-03-30T13:26:29.702909+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:26:09.357648+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:26:09.357648+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:27.784454+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":761,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-30T13:26:28.432776+0000","last_change":"2024-03-30T13:25:20.551619+0000","last_active":"2024-03-30T13:26:28.432776+0000","last_peered":"2024-03-30T13:26:28.432776+0000","last_clean":"2024-03-30T13:26:28.432776+0000","last_became_active":"2024-03-30T13:25:20.549427+0000","last_became_peered":"2024-03-30T13:25:20.549427+0000","last_unstale":"2024-03-30T13:26:28.432776+0000","last_undegraded":"2024-03-30T13:26:28.432776+0000","last_fullsized":"2024-03-30T13:26:28.432776+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:25:15.484555+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:25:15.484555+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:34.070118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":49,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.002419+0000","last_change":"2024-03-30T13:26:25.741558+0000","last_active":"2024-03-30T13:26:31.002419+0000","last_peered":"2024-03-30T13:26:31.002419+0000","last_clean":"2024-03-30T13:26:31.002419+0000","last_became_active":"2024-03-30T13:26:16.578484+0000","last_became_peered":"2024-03-30T13:26:16.578484+0000","last_unstale":"2024-03-30T13:26:31.002419+0000","last_undegraded":"2024-03-30T13:26:31.002419+0000","last_fullsized":"2024-03-30T13:26:31.002419+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:26:25.741212+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:26:25.741212+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:47.925369+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":784,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-30T13:26:28.432906+0000","last_change":"2024-03-30T13:26:09.903819+0000","last_active":"2024-03-30T13:26:28.432906+0000","last_peered":"2024-03-30T13:26:28.432906+0000","last_clean":"2024-03-30T13:26:28.432906+0000","last_became_active":"2024-03-30T13:25:20.556235+0000","last_became_peered":"2024-03-30T13:25:20.556235+0000","last_unstale":"2024-03-30T13:26:28.432906+0000","last_undegraded":"2024-03-30T13:26:28.432906+0000","last_fullsized":"2024-03-30T13:26:28.432906+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:26:09.903568+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_clean_scrub_stamp":"2024-03-30T13:26:09.903568+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:35.258779+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":1066,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-03-30T13:26:26.670798+0000","last_change":"2024-03-30T13:25:58.642430+0000","last_active":"2024-03-30T13:26:26.670798+0000","last_peered":"2024-03-30T13:26:26.670798+0000","last_clean":"2024-03-30T13:26:26.670798+0000","last_became_active":"2024-03-30T13:25:20.550231+0000","last_became_peered":"2024-03-30T13:25:20.550231+0000","last_unstale":"2024-03-30T13:26:26.670798+0000","last_undegraded":"2024-03-30T13:26:26.670798+0000","last_fullsized":"2024-03-30T13:26:26.670798+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:25:58.642180+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:25:58.642180+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:14.896158+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5226911,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":281479874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387769,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59536,"kb_used_data":22328,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94312304,"statfs":{"total":96636764160,"available":96575799296,"internally_reserved":0,"allocated":22863872,"data_stored":18822916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":816,"seq":3504693313552,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68852,"kb_used_data":27048,"kb_used_omap":12,"kb_used_meta":41779,"kb_avail":94302988,"statfs":{"total":96636764160,"available":96566259712,"internally_reserved":0,"allocated":27697152,"data_stored":23669748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12654,"internal_metadata":42782354},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":794,"seq":3410204033050,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59276,"kb_used_data":18752,"kb_used_omap":15,"kb_used_meta":40496,"kb_avail":94312564,"statfs":{"total":96636764160,"available":96576065536,"internally_reserved":0,"allocated":19202048,"data_stored":15174644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15941,"internal_metadata":41468347},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509815,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53036,"kb_used_data":13532,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94318804,"statfs":{"total":96636764160,"available":96582455296,"internally_reserved":0,"allocated":13856768,"data_stored":9822563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":769,"seq":3302829850660,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50432,"kb_used_data":12784,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94321408,"statfs":{"total":96636764160,"available":96585121792,"internally_reserved":0,"allocated":13090816,"data_stored":9070163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10995,"internal_metadata":38524173},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509816,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56060,"kb_used_data":20720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315780,"statfs":{"total":96636764160,"available":96579358720,"internally_reserved":0,"allocated":21217280,"data_stored":17176324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":837,"seq":3594887626758,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58300,"kb_used_data":17264,"kb_used_omap":17,"kb_used_meta":41006,"kb_avail":94313540,"statfs":{"total":96636764160,"available":96577064960,"internally_reserved":0,"allocated":17678336,"data_stored":13637380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17588,"internal_metadata":41990988},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570159,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56484,"kb_used_data":19732,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94315356,"statfs":{"total":96636764160,"available":96578924544,"internally_reserved":0,"allocated":20205568,"data_stored":16157811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:26:32.108 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:26:32.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:32.230+0000 7fb0eee25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:32.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:32.242+0000 7fb0eee25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:32.422 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:26:32.423 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:26:32.440 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1777,"stamp":"2024-03-30T13:26:32.324161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":281479874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":461976,"kb_used_data":152160,"kb_used_omap":75,"kb_used_meta":309620,"kb_avail":754512744,"statfs":{"total":773094113280,"available":772621049856,"internally_reserved":0,"allocated":155811840,"data_stored":123531549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77408,"internal_metadata":317051296},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001088"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1910,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-03-30T13:26:29.703107+0000","last_change":"2024-03-30T13:25:20.550328+0000","last_active":"2024-03-30T13:26:29.703107+0000","last_peered":"2024-03-30T13:26:29.703107+0000","last_clean":"2024-03-30T13:26:29.703107+0000","last_became_active":"2024-03-30T13:25:20.549084+0000","last_became_peered":"2024-03-30T13:25:20.549084+0000","last_unstale":"2024-03-30T13:26:29.703107+0000","last_undegraded":"2024-03-30T13:26:29.703107+0000","last_fullsized":"2024-03-30T13:26:29.703107+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:25:12.543076+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:25:12.543076+0000","last_clean_scrub_stamp":"2024-03-30T13:25:12.543076+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:32.027663+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":13764735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1800,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-30T13:26:27.671887+0000","last_change":"2024-03-30T13:26:19.721841+0000","last_active":"2024-03-30T13:26:27.671887+0000","last_peered":"2024-03-30T13:26:27.671887+0000","last_clean":"2024-03-30T13:26:27.671887+0000","last_became_active":"2024-03-30T13:26:16.583003+0000","last_became_peered":"2024-03-30T13:26:16.583003+0000","last_unstale":"2024-03-30T13:26:27.671887+0000","last_undegraded":"2024-03-30T13:26:27.671887+0000","last_fullsized":"2024-03-30T13:26:27.671887+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:26:19.721595+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:26:19.721595+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:24.825128+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":45,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.002547+0000","last_change":"2024-03-30T13:26:16.646953+0000","last_active":"2024-03-30T13:26:31.002547+0000","last_peered":"2024-03-30T13:26:31.002547+0000","last_clean":"2024-03-30T13:26:31.002547+0000","last_became_active":"2024-03-30T13:26:16.583921+0000","last_became_peered":"2024-03-30T13:26:16.583921+0000","last_unstale":"2024-03-30T13:26:31.002547+0000","last_undegraded":"2024-03-30T13:26:31.002547+0000","last_fullsized":"2024-03-30T13:26:31.002547+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:25:41.165836+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:25:41.165836+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:46.866235+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":51,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.002422+0000","last_change":"2024-03-30T13:26:21.852195+0000","last_active":"2024-03-30T13:26:31.002422+0000","last_peered":"2024-03-30T13:26:31.002422+0000","last_clean":"2024-03-30T13:26:31.002422+0000","last_became_active":"2024-03-30T13:26:16.577250+0000","last_became_peered":"2024-03-30T13:26:16.577250+0000","last_unstale":"2024-03-30T13:26:31.002422+0000","last_undegraded":"2024-03-30T13:26:31.002422+0000","last_fullsized":"2024-03-30T13:26:31.002422+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:26:21.851948+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:26:21.851948+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:41.578957+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":716,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-30T13:26:28.432783+0000","last_change":"2024-03-30T13:25:50.791314+0000","last_active":"2024-03-30T13:26:28.432783+0000","last_peered":"2024-03-30T13:26:28.432783+0000","last_clean":"2024-03-30T13:26:28.432783+0000","last_became_active":"2024-03-30T13:25:20.556165+0000","last_became_peered":"2024-03-30T13:25:20.556165+0000","last_unstale":"2024-03-30T13:26:28.432783+0000","last_undegraded":"2024-03-30T13:26:28.432783+0000","last_fullsized":"2024-03-30T13:26:28.432783+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:25:50.791021+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:25:50.791021+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:01.537702+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1855,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.004313+0000","last_change":"2024-03-30T13:26:25.441051+0000","last_active":"2024-03-30T13:26:31.004313+0000","last_peered":"2024-03-30T13:26:31.004313+0000","last_clean":"2024-03-30T13:26:31.004313+0000","last_became_active":"2024-03-30T13:25:20.551941+0000","last_became_peered":"2024-03-30T13:25:20.551941+0000","last_unstale":"2024-03-30T13:26:31.004313+0000","last_undegraded":"2024-03-30T13:26:31.004313+0000","last_fullsized":"2024-03-30T13:26:31.004313+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:26:25.440798+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:26:25.440798+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:36.431590+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11394981,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":58,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.002420+0000","last_change":"2024-03-30T13:26:17.131560+0000","last_active":"2024-03-30T13:26:31.002420+0000","last_peered":"2024-03-30T13:26:31.002420+0000","last_clean":"2024-03-30T13:26:31.002420+0000","last_became_active":"2024-03-30T13:26:16.583861+0000","last_became_peered":"2024-03-30T13:26:16.583861+0000","last_unstale":"2024-03-30T13:26:31.002420+0000","last_undegraded":"2024-03-30T13:26:31.002420+0000","last_fullsized":"2024-03-30T13:26:31.002420+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:25:56.628976+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:25:56.628976+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:12.425669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1205,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-03-30T13:26:26.670894+0000","last_change":"2024-03-30T13:25:52.600165+0000","last_active":"2024-03-30T13:26:26.670894+0000","last_peered":"2024-03-30T13:26:26.670894+0000","last_clean":"2024-03-30T13:26:26.670894+0000","last_became_active":"2024-03-30T13:25:20.549997+0000","last_became_peered":"2024-03-30T13:25:20.549997+0000","last_unstale":"2024-03-30T13:26:26.670894+0000","last_undegraded":"2024-03-30T13:26:26.670894+0000","last_fullsized":"2024-03-30T13:26:26.670894+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:25:52.599859+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:25:52.599859+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:01.619779+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1976,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.004093+0000","last_change":"2024-03-30T13:26:19.523394+0000","last_active":"2024-03-30T13:26:31.004093+0000","last_peered":"2024-03-30T13:26:31.004093+0000","last_clean":"2024-03-30T13:26:31.004093+0000","last_became_active":"2024-03-30T13:26:16.582448+0000","last_became_peered":"2024-03-30T13:26:16.582448+0000","last_unstale":"2024-03-30T13:26:31.004093+0000","last_undegraded":"2024-03-30T13:26:31.004093+0000","last_fullsized":"2024-03-30T13:26:31.004093+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:26:19.523081+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:26:19.523081+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:39.559039+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":76134720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1602,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-30T13:26:27.671850+0000","last_change":"2024-03-30T13:25:40.607096+0000","last_active":"2024-03-30T13:26:27.671850+0000","last_peered":"2024-03-30T13:26:27.671850+0000","last_clean":"2024-03-30T13:26:27.671850+0000","last_became_active":"2024-03-30T13:24:29.442802+0000","last_became_peered":"2024-03-30T13:24:29.442802+0000","last_unstale":"2024-03-30T13:26:27.671850+0000","last_undegraded":"2024-03-30T13:26:27.671850+0000","last_fullsized":"2024-03-30T13:26:27.671850+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:25:40.606726+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:25:40.606726+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:46.778568+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1925,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-03-30T13:26:29.702870+0000","last_change":"2024-03-30T13:26:01.555623+0000","last_active":"2024-03-30T13:26:29.702870+0000","last_peered":"2024-03-30T13:26:29.702870+0000","last_clean":"2024-03-30T13:26:29.702870+0000","last_became_active":"2024-03-30T13:24:29.447202+0000","last_became_peered":"2024-03-30T13:24:29.447202+0000","last_unstale":"2024-03-30T13:26:29.702870+0000","last_undegraded":"2024-03-30T13:26:29.702870+0000","last_fullsized":"2024-03-30T13:26:29.702870+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:26:01.555372+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:26:01.555372+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:16.483568+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":2019,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-03-30T13:26:29.702909+0000","last_change":"2024-03-30T13:26:09.357933+0000","last_active":"2024-03-30T13:26:29.702909+0000","last_peered":"2024-03-30T13:26:29.702909+0000","last_clean":"2024-03-30T13:26:29.702909+0000","last_became_active":"2024-03-30T13:25:20.549465+0000","last_became_peered":"2024-03-30T13:25:20.549465+0000","last_unstale":"2024-03-30T13:26:29.702909+0000","last_undegraded":"2024-03-30T13:26:29.702909+0000","last_fullsized":"2024-03-30T13:26:29.702909+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:26:09.357648+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:26:09.357648+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:27.784454+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":761,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-30T13:26:28.432776+0000","last_change":"2024-03-30T13:25:20.551619+0000","last_active":"2024-03-30T13:26:28.432776+0000","last_peered":"2024-03-30T13:26:28.432776+0000","last_clean":"2024-03-30T13:26:28.432776+0000","last_became_active":"2024-03-30T13:25:20.549427+0000","last_became_peered":"2024-03-30T13:25:20.549427+0000","last_unstale":"2024-03-30T13:26:28.432776+0000","last_undegraded":"2024-03-30T13:26:28.432776+0000","last_fullsized":"2024-03-30T13:26:28.432776+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:25:15.484555+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:25:15.484555+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:34.070118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":49,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.002419+0000","last_change":"2024-03-30T13:26:25.741558+0000","last_active":"2024-03-30T13:26:31.002419+0000","last_peered":"2024-03-30T13:26:31.002419+0000","last_clean":"2024-03-30T13:26:31.002419+0000","last_became_active":"2024-03-30T13:26:16.578484+0000","last_became_peered":"2024-03-30T13:26:16.578484+0000","last_unstale":"2024-03-30T13:26:31.002419+0000","last_undegraded":"2024-03-30T13:26:31.002419+0000","last_fullsized":"2024-03-30T13:26:31.002419+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:26:25.741212+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:26:25.741212+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:47.925369+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":784,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-30T13:26:28.432906+0000","last_change":"2024-03-30T13:26:09.903819+0000","last_active":"2024-03-30T13:26:28.432906+0000","last_peered":"2024-03-30T13:26:28.432906+0000","last_clean":"2024-03-30T13:26:28.432906+0000","last_became_active":"2024-03-30T13:25:20.556235+0000","last_became_peered":"2024-03-30T13:25:20.556235+0000","last_unstale":"2024-03-30T13:26:28.432906+0000","last_undegraded":"2024-03-30T13:26:28.432906+0000","last_fullsized":"2024-03-30T13:26:28.432906+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:26:09.903568+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_clean_scrub_stamp":"2024-03-30T13:26:09.903568+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:35.258779+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":1066,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-03-30T13:26:26.670798+0000","last_change":"2024-03-30T13:25:58.642430+0000","last_active":"2024-03-30T13:26:26.670798+0000","last_peered":"2024-03-30T13:26:26.670798+0000","last_clean":"2024-03-30T13:26:26.670798+0000","last_became_active":"2024-03-30T13:25:20.550231+0000","last_became_peered":"2024-03-30T13:25:20.550231+0000","last_unstale":"2024-03-30T13:26:26.670798+0000","last_undegraded":"2024-03-30T13:26:26.670798+0000","last_fullsized":"2024-03-30T13:26:26.670798+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:25:58.642180+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:25:58.642180+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:14.896158+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5226911,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":281479874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387769,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59536,"kb_used_data":22328,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94312304,"statfs":{"total":96636764160,"available":96575799296,"internally_reserved":0,"allocated":22863872,"data_stored":18822916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":816,"seq":3504693313552,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68852,"kb_used_data":27048,"kb_used_omap":12,"kb_used_meta":41779,"kb_avail":94302988,"statfs":{"total":96636764160,"available":96566259712,"internally_reserved":0,"allocated":27697152,"data_stored":23669748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12654,"internal_metadata":42782354},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":794,"seq":3410204033050,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59276,"kb_used_data":18752,"kb_used_omap":15,"kb_used_meta":40496,"kb_avail":94312564,"statfs":{"total":96636764160,"available":96576065536,"internally_reserved":0,"allocated":19202048,"data_stored":15174644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15941,"internal_metadata":41468347},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509815,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53036,"kb_used_data":13532,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94318804,"statfs":{"total":96636764160,"available":96582455296,"internally_reserved":0,"allocated":13856768,"data_stored":9822563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":769,"seq":3302829850660,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50432,"kb_used_data":12784,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94321408,"statfs":{"total":96636764160,"available":96585121792,"internally_reserved":0,"allocated":13090816,"data_stored":9070163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10995,"internal_metadata":38524173},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509816,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56060,"kb_used_data":20720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315780,"statfs":{"total":96636764160,"available":96579358720,"internally_reserved":0,"allocated":21217280,"data_stored":17176324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":837,"seq":3594887626758,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58300,"kb_used_data":17264,"kb_used_omap":17,"kb_used_meta":41006,"kb_avail":94313540,"statfs":{"total":96636764160,"available":96577064960,"internally_reserved":0,"allocated":17678336,"data_stored":13637380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17588,"internal_metadata":41990988},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570159,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56484,"kb_used_data":19732,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94315356,"statfs":{"total":96636764160,"available":96578924544,"internally_reserved":0,"allocated":20205568,"data_stored":16157811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:26:32.442 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:26:32.442 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:26:32.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:32.566+0000 7f00f2377640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:32.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:32.578+0000 7f00f2377640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:32.752 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:26:32.753 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":854,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:26:31.710892+0000","last_up_change":"2024-03-30T13:26:11.517361+0000","last_in_change":"2024-03-30T13:26:15.540968+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":41,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":787,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":581,"last_clean_end":763,"up_from":769,"up_thru":833,"down_at":764,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":4049884108},{"type":"v1","addr":"172.21.15.67:6818","nonce":4049884108}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4049884108},{"type":"v1","addr":"172.21.15.67:6821","nonce":4049884108}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":4049884108},{"type":"v1","addr":"172.21.15.67:6828","nonce":4049884108}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":4049884108},{"type":"v1","addr":"172.21.15.67:6825","nonce":4049884108}]},"public_addr":"172.21.15.67:6818/4049884108","cluster_addr":"172.21.15.67:6821/4049884108","heartbeat_back_addr":"172.21.15.67:6828/4049884108","heartbeat_front_addr":"172.21.15.67:6825/4049884108","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":833,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":699,"last_clean_end":738,"up_from":747,"up_thru":838,"down_at":739,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6801","nonce":3161027348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6803","nonce":3161027348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6807","nonce":3161027348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6805","nonce":3161027348}]},"public_addr":"172.21.15.67:6801/3161027348","cluster_addr":"172.21.15.67:6803/3161027348","heartbeat_back_addr":"172.21.15.67:6807/3161027348","heartbeat_front_addr":"172.21.15.67:6805/3161027348","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":677,"last_clean_end":785,"up_from":794,"up_thru":838,"down_at":786,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":167623373},{"type":"v1","addr":"172.21.15.82:6803","nonce":167623373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":167623373},{"type":"v1","addr":"172.21.15.82:6807","nonce":167623373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":167623373},{"type":"v1","addr":"172.21.15.82:6815","nonce":167623373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":167623373},{"type":"v1","addr":"172.21.15.82:6811","nonce":167623373}]},"public_addr":"172.21.15.82:6803/167623373","cluster_addr":"172.21.15.82:6807/167623373","heartbeat_back_addr":"172.21.15.82:6815/167623373","heartbeat_front_addr":"172.21.15.82:6811/167623373","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":515,"last_clean_end":810,"up_from":816,"up_thru":833,"down_at":811,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":1480534728},{"type":"v1","addr":"172.21.15.82:6817","nonce":1480534728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":1480534728},{"type":"v1","addr":"172.21.15.82:6819","nonce":1480534728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":1480534728},{"type":"v1","addr":"172.21.15.82:6823","nonce":1480534728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":1480534728},{"type":"v1","addr":"172.21.15.82:6821","nonce":1480534728}]},"public_addr":"172.21.15.82:6817/1480534728","cluster_addr":"172.21.15.82:6819/1480534728","heartbeat_back_addr":"172.21.15.82:6823/1480534728","heartbeat_front_addr":"172.21.15.82:6821/1480534728","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":629,"last_clean_end":715,"up_from":722,"up_thru":816,"down_at":716,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6801","nonce":326680821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6806","nonce":326680821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6814","nonce":326680821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6810","nonce":326680821}]},"public_addr":"172.21.15.82:6801/326680821","cluster_addr":"172.21.15.82:6806/326680821","heartbeat_back_addr":"172.21.15.82:6814/326680821","heartbeat_front_addr":"172.21.15.82:6810/326680821","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":650,"last_clean_end":832,"up_from":837,"up_thru":838,"down_at":833,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":876600018},{"type":"v1","addr":"172.21.15.82:6825","nonce":876600018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":876600018},{"type":"v1","addr":"172.21.15.82:6827","nonce":876600018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":876600018},{"type":"v1","addr":"172.21.15.82:6831","nonce":876600018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":876600018},{"type":"v1","addr":"172.21.15.82:6829","nonce":876600018}]},"public_addr":"172.21.15.82:6825/876600018","cluster_addr":"172.21.15.82:6827/876600018","heartbeat_back_addr":"172.21.15.82:6831/876600018","heartbeat_front_addr":"172.21.15.82:6829/876600018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:25:34.584003+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:26:28.993270+0000","dead_epoch":764},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:26:23.044916+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:25:27.862088+0000","dead_epoch":739},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:25:53.055644+0000","dead_epoch":786},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:26:27.132139+0000","dead_epoch":811},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:26:10.578437+0000","dead_epoch":716},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:25:21.205496+0000","dead_epoch":833}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:26:32.772 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 min_size is 2 2024-03-30T13:26:32.772 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get unique_pool_0 erasure_code_profile 2024-03-30T13:26:32.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:32.890+0000 7f55641ab640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:32.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:32.906+0000 7f55641ab640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:33.078 INFO:teuthology.orchestra.run.smithi067.stdout:erasure_code_profile: teuthologyprofile 2024-03-30T13:26:33.098 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile get teuthologyprofile --format=json 2024-03-30T13:26:33.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:33.218+0000 7f2812635640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:33.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:33.234+0000 7f2812635640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:33.418 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:26:33.418 INFO:teuthology.orchestra.run.smithi067.stdout:{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"} 2024-03-30T13:26:33.438 INFO:tasks.thrashosds.thrasher:pool unique_pool_0 local_k=2 local_m=1 2024-03-30T13:26:33.438 INFO:tasks.thrashosds.thrasher:setting k=2 from previous 0 2024-03-30T13:26:33.438 INFO:tasks.thrashosds.thrasher:setting m=1 from previous 99 2024-03-30T13:26:33.438 INFO:tasks.thrashosds.thrasher:using k=2, m=1 2024-03-30T13:26:33.439 INFO:tasks.thrashosds.thrasher:dead_osds=[], live_osds=[0, 2, 6, 3, 1, 4, 5, 7] 2024-03-30T13:26:33.439 INFO:tasks.thrashosds.thrasher:chose to kill 1 OSDs 2024-03-30T13:26:33.439 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:26:33.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:33.550+0000 7fcb35998640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:33.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:33.562+0000 7fcb35998640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:33.729 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:26:33.730 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:26:33.747 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1777,"stamp":"2024-03-30T13:26:32.324161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":281479874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":461976,"kb_used_data":152160,"kb_used_omap":75,"kb_used_meta":309620,"kb_avail":754512744,"statfs":{"total":773094113280,"available":772621049856,"internally_reserved":0,"allocated":155811840,"data_stored":123531549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77408,"internal_metadata":317051296},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001088"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1910,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-03-30T13:26:29.703107+0000","last_change":"2024-03-30T13:25:20.550328+0000","last_active":"2024-03-30T13:26:29.703107+0000","last_peered":"2024-03-30T13:26:29.703107+0000","last_clean":"2024-03-30T13:26:29.703107+0000","last_became_active":"2024-03-30T13:25:20.549084+0000","last_became_peered":"2024-03-30T13:25:20.549084+0000","last_unstale":"2024-03-30T13:26:29.703107+0000","last_undegraded":"2024-03-30T13:26:29.703107+0000","last_fullsized":"2024-03-30T13:26:29.703107+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:25:12.543076+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:25:12.543076+0000","last_clean_scrub_stamp":"2024-03-30T13:25:12.543076+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:32.027663+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":13764735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1800,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-30T13:26:27.671887+0000","last_change":"2024-03-30T13:26:19.721841+0000","last_active":"2024-03-30T13:26:27.671887+0000","last_peered":"2024-03-30T13:26:27.671887+0000","last_clean":"2024-03-30T13:26:27.671887+0000","last_became_active":"2024-03-30T13:26:16.583003+0000","last_became_peered":"2024-03-30T13:26:16.583003+0000","last_unstale":"2024-03-30T13:26:27.671887+0000","last_undegraded":"2024-03-30T13:26:27.671887+0000","last_fullsized":"2024-03-30T13:26:27.671887+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:26:19.721595+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:26:19.721595+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:24.825128+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":45,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.002547+0000","last_change":"2024-03-30T13:26:16.646953+0000","last_active":"2024-03-30T13:26:31.002547+0000","last_peered":"2024-03-30T13:26:31.002547+0000","last_clean":"2024-03-30T13:26:31.002547+0000","last_became_active":"2024-03-30T13:26:16.583921+0000","last_became_peered":"2024-03-30T13:26:16.583921+0000","last_unstale":"2024-03-30T13:26:31.002547+0000","last_undegraded":"2024-03-30T13:26:31.002547+0000","last_fullsized":"2024-03-30T13:26:31.002547+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:25:41.165836+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:25:41.165836+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:46.866235+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":51,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.002422+0000","last_change":"2024-03-30T13:26:21.852195+0000","last_active":"2024-03-30T13:26:31.002422+0000","last_peered":"2024-03-30T13:26:31.002422+0000","last_clean":"2024-03-30T13:26:31.002422+0000","last_became_active":"2024-03-30T13:26:16.577250+0000","last_became_peered":"2024-03-30T13:26:16.577250+0000","last_unstale":"2024-03-30T13:26:31.002422+0000","last_undegraded":"2024-03-30T13:26:31.002422+0000","last_fullsized":"2024-03-30T13:26:31.002422+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:26:21.851948+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:26:21.851948+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:41.578957+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":716,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-30T13:26:28.432783+0000","last_change":"2024-03-30T13:25:50.791314+0000","last_active":"2024-03-30T13:26:28.432783+0000","last_peered":"2024-03-30T13:26:28.432783+0000","last_clean":"2024-03-30T13:26:28.432783+0000","last_became_active":"2024-03-30T13:25:20.556165+0000","last_became_peered":"2024-03-30T13:25:20.556165+0000","last_unstale":"2024-03-30T13:26:28.432783+0000","last_undegraded":"2024-03-30T13:26:28.432783+0000","last_fullsized":"2024-03-30T13:26:28.432783+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:25:50.791021+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:25:50.791021+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:01.537702+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1855,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.004313+0000","last_change":"2024-03-30T13:26:25.441051+0000","last_active":"2024-03-30T13:26:31.004313+0000","last_peered":"2024-03-30T13:26:31.004313+0000","last_clean":"2024-03-30T13:26:31.004313+0000","last_became_active":"2024-03-30T13:25:20.551941+0000","last_became_peered":"2024-03-30T13:25:20.551941+0000","last_unstale":"2024-03-30T13:26:31.004313+0000","last_undegraded":"2024-03-30T13:26:31.004313+0000","last_fullsized":"2024-03-30T13:26:31.004313+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:26:25.440798+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:26:25.440798+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:36.431590+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11394981,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":58,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.002420+0000","last_change":"2024-03-30T13:26:17.131560+0000","last_active":"2024-03-30T13:26:31.002420+0000","last_peered":"2024-03-30T13:26:31.002420+0000","last_clean":"2024-03-30T13:26:31.002420+0000","last_became_active":"2024-03-30T13:26:16.583861+0000","last_became_peered":"2024-03-30T13:26:16.583861+0000","last_unstale":"2024-03-30T13:26:31.002420+0000","last_undegraded":"2024-03-30T13:26:31.002420+0000","last_fullsized":"2024-03-30T13:26:31.002420+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:25:56.628976+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:25:56.628976+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:12.425669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1205,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-03-30T13:26:26.670894+0000","last_change":"2024-03-30T13:25:52.600165+0000","last_active":"2024-03-30T13:26:26.670894+0000","last_peered":"2024-03-30T13:26:26.670894+0000","last_clean":"2024-03-30T13:26:26.670894+0000","last_became_active":"2024-03-30T13:25:20.549997+0000","last_became_peered":"2024-03-30T13:25:20.549997+0000","last_unstale":"2024-03-30T13:26:26.670894+0000","last_undegraded":"2024-03-30T13:26:26.670894+0000","last_fullsized":"2024-03-30T13:26:26.670894+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:25:52.599859+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:25:52.599859+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:01.619779+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1976,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.004093+0000","last_change":"2024-03-30T13:26:19.523394+0000","last_active":"2024-03-30T13:26:31.004093+0000","last_peered":"2024-03-30T13:26:31.004093+0000","last_clean":"2024-03-30T13:26:31.004093+0000","last_became_active":"2024-03-30T13:26:16.582448+0000","last_became_peered":"2024-03-30T13:26:16.582448+0000","last_unstale":"2024-03-30T13:26:31.004093+0000","last_undegraded":"2024-03-30T13:26:31.004093+0000","last_fullsized":"2024-03-30T13:26:31.004093+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:26:19.523081+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:26:19.523081+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:39.559039+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":76134720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1602,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-30T13:26:27.671850+0000","last_change":"2024-03-30T13:25:40.607096+0000","last_active":"2024-03-30T13:26:27.671850+0000","last_peered":"2024-03-30T13:26:27.671850+0000","last_clean":"2024-03-30T13:26:27.671850+0000","last_became_active":"2024-03-30T13:24:29.442802+0000","last_became_peered":"2024-03-30T13:24:29.442802+0000","last_unstale":"2024-03-30T13:26:27.671850+0000","last_undegraded":"2024-03-30T13:26:27.671850+0000","last_fullsized":"2024-03-30T13:26:27.671850+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:25:40.606726+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:25:40.606726+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:46.778568+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1925,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-03-30T13:26:29.702870+0000","last_change":"2024-03-30T13:26:01.555623+0000","last_active":"2024-03-30T13:26:29.702870+0000","last_peered":"2024-03-30T13:26:29.702870+0000","last_clean":"2024-03-30T13:26:29.702870+0000","last_became_active":"2024-03-30T13:24:29.447202+0000","last_became_peered":"2024-03-30T13:24:29.447202+0000","last_unstale":"2024-03-30T13:26:29.702870+0000","last_undegraded":"2024-03-30T13:26:29.702870+0000","last_fullsized":"2024-03-30T13:26:29.702870+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:26:01.555372+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:26:01.555372+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:16.483568+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":2019,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-03-30T13:26:29.702909+0000","last_change":"2024-03-30T13:26:09.357933+0000","last_active":"2024-03-30T13:26:29.702909+0000","last_peered":"2024-03-30T13:26:29.702909+0000","last_clean":"2024-03-30T13:26:29.702909+0000","last_became_active":"2024-03-30T13:25:20.549465+0000","last_became_peered":"2024-03-30T13:25:20.549465+0000","last_unstale":"2024-03-30T13:26:29.702909+0000","last_undegraded":"2024-03-30T13:26:29.702909+0000","last_fullsized":"2024-03-30T13:26:29.702909+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:26:09.357648+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:26:09.357648+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:27.784454+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":761,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-30T13:26:28.432776+0000","last_change":"2024-03-30T13:25:20.551619+0000","last_active":"2024-03-30T13:26:28.432776+0000","last_peered":"2024-03-30T13:26:28.432776+0000","last_clean":"2024-03-30T13:26:28.432776+0000","last_became_active":"2024-03-30T13:25:20.549427+0000","last_became_peered":"2024-03-30T13:25:20.549427+0000","last_unstale":"2024-03-30T13:26:28.432776+0000","last_undegraded":"2024-03-30T13:26:28.432776+0000","last_fullsized":"2024-03-30T13:26:28.432776+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:25:15.484555+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:25:15.484555+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:34.070118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":49,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.002419+0000","last_change":"2024-03-30T13:26:25.741558+0000","last_active":"2024-03-30T13:26:31.002419+0000","last_peered":"2024-03-30T13:26:31.002419+0000","last_clean":"2024-03-30T13:26:31.002419+0000","last_became_active":"2024-03-30T13:26:16.578484+0000","last_became_peered":"2024-03-30T13:26:16.578484+0000","last_unstale":"2024-03-30T13:26:31.002419+0000","last_undegraded":"2024-03-30T13:26:31.002419+0000","last_fullsized":"2024-03-30T13:26:31.002419+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:26:25.741212+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:26:25.741212+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:47.925369+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":784,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-03-30T13:26:28.432906+0000","last_change":"2024-03-30T13:26:09.903819+0000","last_active":"2024-03-30T13:26:28.432906+0000","last_peered":"2024-03-30T13:26:28.432906+0000","last_clean":"2024-03-30T13:26:28.432906+0000","last_became_active":"2024-03-30T13:25:20.556235+0000","last_became_peered":"2024-03-30T13:25:20.556235+0000","last_unstale":"2024-03-30T13:26:28.432906+0000","last_undegraded":"2024-03-30T13:26:28.432906+0000","last_fullsized":"2024-03-30T13:26:28.432906+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:26:09.903568+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_clean_scrub_stamp":"2024-03-30T13:26:09.903568+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:35.258779+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":1066,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-03-30T13:26:26.670798+0000","last_change":"2024-03-30T13:25:58.642430+0000","last_active":"2024-03-30T13:26:26.670798+0000","last_peered":"2024-03-30T13:26:26.670798+0000","last_clean":"2024-03-30T13:26:26.670798+0000","last_became_active":"2024-03-30T13:25:20.550231+0000","last_became_peered":"2024-03-30T13:25:20.550231+0000","last_unstale":"2024-03-30T13:26:26.670798+0000","last_undegraded":"2024-03-30T13:26:26.670798+0000","last_fullsized":"2024-03-30T13:26:26.670798+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:25:58.642180+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:25:58.642180+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:14.896158+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5226911,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":281479874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387769,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59536,"kb_used_data":22328,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94312304,"statfs":{"total":96636764160,"available":96575799296,"internally_reserved":0,"allocated":22863872,"data_stored":18822916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":816,"seq":3504693313552,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68852,"kb_used_data":27048,"kb_used_omap":12,"kb_used_meta":41779,"kb_avail":94302988,"statfs":{"total":96636764160,"available":96566259712,"internally_reserved":0,"allocated":27697152,"data_stored":23669748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12654,"internal_metadata":42782354},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":794,"seq":3410204033050,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59276,"kb_used_data":18752,"kb_used_omap":15,"kb_used_meta":40496,"kb_avail":94312564,"statfs":{"total":96636764160,"available":96576065536,"internally_reserved":0,"allocated":19202048,"data_stored":15174644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15941,"internal_metadata":41468347},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509815,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53036,"kb_used_data":13532,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94318804,"statfs":{"total":96636764160,"available":96582455296,"internally_reserved":0,"allocated":13856768,"data_stored":9822563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":769,"seq":3302829850660,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50432,"kb_used_data":12784,"kb_used_omap":10,"kb_used_meta":37621,"kb_avail":94321408,"statfs":{"total":96636764160,"available":96585121792,"internally_reserved":0,"allocated":13090816,"data_stored":9070163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10995,"internal_metadata":38524173},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509816,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56060,"kb_used_data":20720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315780,"statfs":{"total":96636764160,"available":96579358720,"internally_reserved":0,"allocated":21217280,"data_stored":17176324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":837,"seq":3594887626758,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58300,"kb_used_data":17264,"kb_used_omap":17,"kb_used_meta":41006,"kb_avail":94313540,"statfs":{"total":96636764160,"available":96577064960,"internally_reserved":0,"allocated":17678336,"data_stored":13637380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17588,"internal_metadata":41990988},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570159,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56484,"kb_used_data":19732,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94315356,"statfs":{"total":96636764160,"available":96578924544,"internally_reserved":0,"allocated":20205568,"data_stored":16157811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:26:33.748 INFO:tasks.thrashosds.thrasher:Choosing PG 1.3 with acting set [1, 6, 5] 2024-03-30T13:26:33.748 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 2, 6, 3, 1, 4, 5, 7] 2024-03-30T13:26:33.749 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-30T13:26:33.749 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:26:33.847 INFO:tasks.ceph.osd.1:Stopped 2024-03-30T13:26:33.848 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 2, 6, 3, 1, 4, 5, 7] 2024-03-30T13:26:33.848 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 1 2024-03-30T13:26:33.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:33.966+0000 7f4886e52640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:33.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:33.982+0000 7f4886e52640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:34.156 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:26:34.158+0000 7fbb38f08640 -1 mon.a@0(leader).osd e854 definitely_dead 0 2024-03-30T13:26:34.742 INFO:teuthology.orchestra.run.smithi067.stderr:marked out osd.1. 2024-03-30T13:26:34.772 INFO:tasks.thrashosds.thrasher:dead_osds=[1], live_osds=[0, 2, 6, 3, 4, 5, 7] 2024-03-30T13:26:34.772 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:26:34.772 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:26:34.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:34.890+0000 7f9c3780e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:34.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:26:34.902+0000 7f9c3780e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:26:35.075 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:26:35.076 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:26:35.094 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1779,"stamp":"2024-03-30T13:26:34.745816+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":281479874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":45,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":411652,"kb_used_data":139472,"kb_used_omap":64,"kb_used_meta":271999,"kb_avail":660191228,"statfs":{"total":676457349120,"available":676035817472,"internally_reserved":0,"allocated":142819328,"data_stored":114505382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":66413,"internal_metadata":278527123},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.044836"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1910,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-03-30T13:26:29.703107+0000","last_change":"2024-03-30T13:25:20.550328+0000","last_active":"2024-03-30T13:26:29.703107+0000","last_peered":"2024-03-30T13:26:29.703107+0000","last_clean":"2024-03-30T13:26:29.703107+0000","last_became_active":"2024-03-30T13:25:20.549084+0000","last_became_peered":"2024-03-30T13:25:20.549084+0000","last_unstale":"2024-03-30T13:26:29.703107+0000","last_undegraded":"2024-03-30T13:26:29.703107+0000","last_fullsized":"2024-03-30T13:26:29.703107+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:25:12.543076+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:25:12.543076+0000","last_clean_scrub_stamp":"2024-03-30T13:25:12.543076+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:32.027663+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":13764735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1807,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.720245+0000","last_change":"2024-03-30T13:26:19.721841+0000","last_active":"2024-03-30T13:26:31.720245+0000","last_peered":"2024-03-30T13:26:31.720245+0000","last_clean":"2024-03-30T13:26:31.720245+0000","last_became_active":"2024-03-30T13:26:16.583003+0000","last_became_peered":"2024-03-30T13:26:16.583003+0000","last_unstale":"2024-03-30T13:26:31.720245+0000","last_undegraded":"2024-03-30T13:26:31.720245+0000","last_fullsized":"2024-03-30T13:26:31.720245+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:26:19.721595+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:26:19.721595+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:24.825128+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":45,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.002547+0000","last_change":"2024-03-30T13:26:16.646953+0000","last_active":"2024-03-30T13:26:31.002547+0000","last_peered":"2024-03-30T13:26:31.002547+0000","last_clean":"2024-03-30T13:26:31.002547+0000","last_became_active":"2024-03-30T13:26:16.583921+0000","last_became_peered":"2024-03-30T13:26:16.583921+0000","last_unstale":"2024-03-30T13:26:31.002547+0000","last_undegraded":"2024-03-30T13:26:31.002547+0000","last_fullsized":"2024-03-30T13:26:31.002547+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:25:41.165836+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:25:41.165836+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:46.866235+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":23445468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":51,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.002422+0000","last_change":"2024-03-30T13:26:21.852195+0000","last_active":"2024-03-30T13:26:31.002422+0000","last_peered":"2024-03-30T13:26:31.002422+0000","last_clean":"2024-03-30T13:26:31.002422+0000","last_became_active":"2024-03-30T13:26:16.577250+0000","last_became_peered":"2024-03-30T13:26:16.577250+0000","last_unstale":"2024-03-30T13:26:31.002422+0000","last_undegraded":"2024-03-30T13:26:31.002422+0000","last_fullsized":"2024-03-30T13:26:31.002422+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:26:21.851948+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:26:21.851948+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:41.578957+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":723,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.717666+0000","last_change":"2024-03-30T13:25:50.791314+0000","last_active":"2024-03-30T13:26:31.717666+0000","last_peered":"2024-03-30T13:26:31.717666+0000","last_clean":"2024-03-30T13:26:31.717666+0000","last_became_active":"2024-03-30T13:25:20.556165+0000","last_became_peered":"2024-03-30T13:25:20.556165+0000","last_unstale":"2024-03-30T13:26:31.717666+0000","last_undegraded":"2024-03-30T13:26:31.717666+0000","last_fullsized":"2024-03-30T13:26:31.717666+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:25:50.791021+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:25:50.791021+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:01.537702+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1855,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.004313+0000","last_change":"2024-03-30T13:26:25.441051+0000","last_active":"2024-03-30T13:26:31.004313+0000","last_peered":"2024-03-30T13:26:31.004313+0000","last_clean":"2024-03-30T13:26:31.004313+0000","last_became_active":"2024-03-30T13:25:20.551941+0000","last_became_peered":"2024-03-30T13:25:20.551941+0000","last_unstale":"2024-03-30T13:26:31.004313+0000","last_undegraded":"2024-03-30T13:26:31.004313+0000","last_fullsized":"2024-03-30T13:26:31.004313+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:26:25.440798+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:26:25.440798+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:36.431590+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11394981,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":58,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.002420+0000","last_change":"2024-03-30T13:26:17.131560+0000","last_active":"2024-03-30T13:26:31.002420+0000","last_peered":"2024-03-30T13:26:31.002420+0000","last_clean":"2024-03-30T13:26:31.002420+0000","last_became_active":"2024-03-30T13:26:16.583861+0000","last_became_peered":"2024-03-30T13:26:16.583861+0000","last_unstale":"2024-03-30T13:26:31.002420+0000","last_undegraded":"2024-03-30T13:26:31.002420+0000","last_fullsized":"2024-03-30T13:26:31.002420+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:25:56.628976+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:25:56.628976+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:12.425669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1215,"reported_epoch":854,"state":"stale+active+clean","last_fresh":"2024-03-30T13:26:31.885955+0000","last_change":"2024-03-30T13:25:52.600165+0000","last_active":"2024-03-30T13:26:31.885955+0000","last_peered":"2024-03-30T13:26:31.885955+0000","last_clean":"2024-03-30T13:26:31.885955+0000","last_became_active":"2024-03-30T13:25:20.549997+0000","last_became_peered":"2024-03-30T13:25:20.549997+0000","last_unstale":"2024-03-30T13:26:34.745852+0000","last_undegraded":"2024-03-30T13:26:31.885955+0000","last_fullsized":"2024-03-30T13:26:31.885955+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:25:52.599859+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:25:52.599859+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:01.619779+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":66101040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1976,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.004093+0000","last_change":"2024-03-30T13:26:19.523394+0000","last_active":"2024-03-30T13:26:31.004093+0000","last_peered":"2024-03-30T13:26:31.004093+0000","last_clean":"2024-03-30T13:26:31.004093+0000","last_became_active":"2024-03-30T13:26:16.582448+0000","last_became_peered":"2024-03-30T13:26:16.582448+0000","last_unstale":"2024-03-30T13:26:31.004093+0000","last_undegraded":"2024-03-30T13:26:31.004093+0000","last_fullsized":"2024-03-30T13:26:31.004093+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:26:19.523081+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:26:19.523081+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:39.559039+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":76134720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1610,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.720104+0000","last_change":"2024-03-30T13:25:40.607096+0000","last_active":"2024-03-30T13:26:31.720104+0000","last_peered":"2024-03-30T13:26:31.720104+0000","last_clean":"2024-03-30T13:26:31.720104+0000","last_became_active":"2024-03-30T13:24:29.442802+0000","last_became_peered":"2024-03-30T13:24:29.442802+0000","last_unstale":"2024-03-30T13:26:31.720104+0000","last_undegraded":"2024-03-30T13:26:31.720104+0000","last_fullsized":"2024-03-30T13:26:31.720104+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:25:40.606726+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:25:40.606726+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:46.778568+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1925,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-03-30T13:26:29.702870+0000","last_change":"2024-03-30T13:26:01.555623+0000","last_active":"2024-03-30T13:26:29.702870+0000","last_peered":"2024-03-30T13:26:29.702870+0000","last_clean":"2024-03-30T13:26:29.702870+0000","last_became_active":"2024-03-30T13:24:29.447202+0000","last_became_peered":"2024-03-30T13:24:29.447202+0000","last_unstale":"2024-03-30T13:26:29.702870+0000","last_undegraded":"2024-03-30T13:26:29.702870+0000","last_fullsized":"2024-03-30T13:26:29.702870+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:26:01.555372+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:26:01.555372+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:16.483568+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":2019,"reported_epoch":852,"state":"active+clean","last_fresh":"2024-03-30T13:26:29.702909+0000","last_change":"2024-03-30T13:26:09.357933+0000","last_active":"2024-03-30T13:26:29.702909+0000","last_peered":"2024-03-30T13:26:29.702909+0000","last_clean":"2024-03-30T13:26:29.702909+0000","last_became_active":"2024-03-30T13:25:20.549465+0000","last_became_peered":"2024-03-30T13:25:20.549465+0000","last_unstale":"2024-03-30T13:26:29.702909+0000","last_undegraded":"2024-03-30T13:26:29.702909+0000","last_fullsized":"2024-03-30T13:26:29.702909+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:26:09.357648+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:26:09.357648+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:27.784454+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":768,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.716998+0000","last_change":"2024-03-30T13:25:20.551619+0000","last_active":"2024-03-30T13:26:31.716998+0000","last_peered":"2024-03-30T13:26:31.716998+0000","last_clean":"2024-03-30T13:26:31.716998+0000","last_became_active":"2024-03-30T13:25:20.549427+0000","last_became_peered":"2024-03-30T13:25:20.549427+0000","last_unstale":"2024-03-30T13:26:31.716998+0000","last_undegraded":"2024-03-30T13:26:31.716998+0000","last_fullsized":"2024-03-30T13:26:31.716998+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:25:15.484555+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:25:15.484555+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:26:34.070118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":49,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.002419+0000","last_change":"2024-03-30T13:26:25.741558+0000","last_active":"2024-03-30T13:26:31.002419+0000","last_peered":"2024-03-30T13:26:31.002419+0000","last_clean":"2024-03-30T13:26:31.002419+0000","last_became_active":"2024-03-30T13:26:16.578484+0000","last_became_peered":"2024-03-30T13:26:16.578484+0000","last_unstale":"2024-03-30T13:26:31.002419+0000","last_undegraded":"2024-03-30T13:26:31.002419+0000","last_fullsized":"2024-03-30T13:26:31.002419+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:26:25.741212+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:26:25.741212+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:47.925369+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":791,"reported_epoch":854,"state":"active+clean","last_fresh":"2024-03-30T13:26:31.716998+0000","last_change":"2024-03-30T13:26:09.903819+0000","last_active":"2024-03-30T13:26:31.716998+0000","last_peered":"2024-03-30T13:26:31.716998+0000","last_clean":"2024-03-30T13:26:31.716998+0000","last_became_active":"2024-03-30T13:25:20.556235+0000","last_became_peered":"2024-03-30T13:25:20.556235+0000","last_unstale":"2024-03-30T13:26:31.716998+0000","last_undegraded":"2024-03-30T13:26:31.716998+0000","last_fullsized":"2024-03-30T13:26:31.716998+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:26:09.903568+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_clean_scrub_stamp":"2024-03-30T13:26:09.903568+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:35.258779+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":1076,"reported_epoch":854,"state":"stale+active+clean","last_fresh":"2024-03-30T13:26:31.885973+0000","last_change":"2024-03-30T13:25:58.642430+0000","last_active":"2024-03-30T13:26:31.885973+0000","last_peered":"2024-03-30T13:26:31.885973+0000","last_clean":"2024-03-30T13:26:31.885973+0000","last_became_active":"2024-03-30T13:25:20.550231+0000","last_became_peered":"2024-03-30T13:25:20.550231+0000","last_unstale":"2024-03-30T13:26:34.745857+0000","last_undegraded":"2024-03-30T13:26:31.885973+0000","last_fullsized":"2024-03-30T13:26:31.885973+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:25:58.642180+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:25:58.642180+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:14.896158+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5226911,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":281479874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387769,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59536,"kb_used_data":22328,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94312304,"statfs":{"total":96636764160,"available":96575799296,"internally_reserved":0,"allocated":22863872,"data_stored":18822916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":816,"seq":3504693313553,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68908,"kb_used_data":27096,"kb_used_omap":12,"kb_used_meta":41779,"kb_avail":94302932,"statfs":{"total":96636764160,"available":96566202368,"internally_reserved":0,"allocated":27746304,"data_stored":23691746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12654,"internal_metadata":42782354},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":794,"seq":3410204033051,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59328,"kb_used_data":18800,"kb_used_omap":15,"kb_used_meta":40496,"kb_avail":94312512,"statfs":{"total":96636764160,"available":96576012288,"internally_reserved":0,"allocated":19251200,"data_stored":15196642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15941,"internal_metadata":41468347},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509815,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53036,"kb_used_data":13532,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94318804,"statfs":{"total":96636764160,"available":96582455296,"internally_reserved":0,"allocated":13856768,"data_stored":9822563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509816,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56060,"kb_used_data":20720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315780,"statfs":{"total":96636764160,"available":96579358720,"internally_reserved":0,"allocated":21217280,"data_stored":17176324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":837,"seq":3594887626758,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58300,"kb_used_data":17264,"kb_used_omap":17,"kb_used_meta":41006,"kb_avail":94313540,"statfs":{"total":96636764160,"available":96577064960,"internally_reserved":0,"allocated":17678336,"data_stored":13637380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17588,"internal_metadata":41990988},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570159,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56484,"kb_used_data":19732,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94315356,"statfs":{"total":96636764160,"available":96578924544,"internally_reserved":0,"allocated":20205568,"data_stored":16157811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:26:35.095 INFO:tasks.thrashosds.thrasher:not all PGs are active or peered 2024-03-30T13:27:00.096 INFO:tasks.ceph.ceph_manager.ceph:checking for active or peered 2024-03-30T13:27:00.096 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:27:00.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:00.225+0000 7f03cbf19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:00.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:00.237+0000 7f03cbf19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:00.412 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:27:00.413 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:27:00.431 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1796,"stamp":"2024-03-30T13:27:00.332421+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":281950470,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":417780,"kb_used_data":145612,"kb_used_omap":64,"kb_used_meta":271999,"kb_avail":660185100,"statfs":{"total":676457349120,"available":676029542400,"internally_reserved":0,"allocated":149106688,"data_stored":120488588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":66413,"internal_metadata":278527123},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.932682"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1932,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-03-30T13:26:49.612302+0000","last_change":"2024-03-30T13:26:34.531020+0000","last_active":"2024-03-30T13:26:49.612302+0000","last_peered":"2024-03-30T13:26:49.612302+0000","last_clean":"2024-03-30T13:26:49.612302+0000","last_became_active":"2024-03-30T13:25:20.549084+0000","last_became_peered":"2024-03-30T13:25:20.549084+0000","last_unstale":"2024-03-30T13:26:49.612302+0000","last_undegraded":"2024-03-30T13:26:49.612302+0000","last_fullsized":"2024-03-30T13:26:49.612302+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:26:34.530597+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:25:12.543076+0000","last_clean_scrub_stamp":"2024-03-30T13:26:34.530597+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:35.601883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":13764735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1818,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-03-30T13:26:49.402074+0000","last_change":"2024-03-30T13:26:19.721841+0000","last_active":"2024-03-30T13:26:49.402074+0000","last_peered":"2024-03-30T13:26:49.402074+0000","last_clean":"2024-03-30T13:26:49.402074+0000","last_became_active":"2024-03-30T13:26:16.583003+0000","last_became_peered":"2024-03-30T13:26:16.583003+0000","last_unstale":"2024-03-30T13:26:49.402074+0000","last_undegraded":"2024-03-30T13:26:49.402074+0000","last_fullsized":"2024-03-30T13:26:49.402074+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:26:19.721595+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:26:19.721595+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:24.825128+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":79,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-03-30T13:26:49.611497+0000","last_change":"2024-03-30T13:26:47.672589+0000","last_active":"2024-03-30T13:26:49.611497+0000","last_peered":"2024-03-30T13:26:49.611497+0000","last_clean":"2024-03-30T13:26:49.611497+0000","last_became_active":"2024-03-30T13:26:35.760877+0000","last_became_peered":"2024-03-30T13:26:35.760877+0000","last_unstale":"2024-03-30T13:26:49.611497+0000","last_undegraded":"2024-03-30T13:26:49.611497+0000","last_fullsized":"2024-03-30T13:26:49.611497+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:26:47.672328+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:26:47.672328+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:58.588932+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":63,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-03-30T13:26:49.611377+0000","last_change":"2024-03-30T13:26:21.852195+0000","last_active":"2024-03-30T13:26:49.611377+0000","last_peered":"2024-03-30T13:26:49.611377+0000","last_clean":"2024-03-30T13:26:49.611377+0000","last_became_active":"2024-03-30T13:26:16.577250+0000","last_became_peered":"2024-03-30T13:26:16.577250+0000","last_unstale":"2024-03-30T13:26:49.611377+0000","last_undegraded":"2024-03-30T13:26:49.611377+0000","last_fullsized":"2024-03-30T13:26:49.611377+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:26:21.851948+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:26:21.851948+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:41.578957+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":733,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-03-30T13:26:49.612312+0000","last_change":"2024-03-30T13:25:50.791314+0000","last_active":"2024-03-30T13:26:49.612312+0000","last_peered":"2024-03-30T13:26:49.612312+0000","last_clean":"2024-03-30T13:26:49.612312+0000","last_became_active":"2024-03-30T13:25:20.556165+0000","last_became_peered":"2024-03-30T13:25:20.556165+0000","last_unstale":"2024-03-30T13:26:49.612312+0000","last_undegraded":"2024-03-30T13:26:49.612312+0000","last_fullsized":"2024-03-30T13:26:49.612312+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:25:50.791021+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:25:50.791021+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:01.537702+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1867,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-03-30T13:26:49.612192+0000","last_change":"2024-03-30T13:26:25.441051+0000","last_active":"2024-03-30T13:26:49.612192+0000","last_peered":"2024-03-30T13:26:49.612192+0000","last_clean":"2024-03-30T13:26:49.612192+0000","last_became_active":"2024-03-30T13:25:20.551941+0000","last_became_peered":"2024-03-30T13:25:20.551941+0000","last_unstale":"2024-03-30T13:26:49.612192+0000","last_undegraded":"2024-03-30T13:26:49.612192+0000","last_fullsized":"2024-03-30T13:26:49.612192+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:26:25.440798+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:26:25.440798+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:36.431590+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11394981,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":70,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-03-30T13:26:49.611479+0000","last_change":"2024-03-30T13:26:17.131560+0000","last_active":"2024-03-30T13:26:49.611479+0000","last_peered":"2024-03-30T13:26:49.611479+0000","last_clean":"2024-03-30T13:26:49.611479+0000","last_became_active":"2024-03-30T13:26:16.583861+0000","last_became_peered":"2024-03-30T13:26:16.583861+0000","last_unstale":"2024-03-30T13:26:49.611479+0000","last_undegraded":"2024-03-30T13:26:49.611479+0000","last_fullsized":"2024-03-30T13:26:49.611479+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:25:56.628976+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:25:56.628976+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:12.425669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":25,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-03-30T13:26:49.402005+0000","last_change":"2024-03-30T13:26:35.822398+0000","last_active":"2024-03-30T13:26:49.402005+0000","last_peered":"2024-03-30T13:26:49.402005+0000","last_clean":"2024-03-30T13:26:49.402005+0000","last_became_active":"2024-03-30T13:26:35.758908+0000","last_became_peered":"2024-03-30T13:26:35.758908+0000","last_unstale":"2024-03-30T13:26:49.402005+0000","last_undegraded":"2024-03-30T13:26:49.402005+0000","last_fullsized":"2024-03-30T13:26:49.402005+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:25:52.599859+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:25:52.599859+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:22.047355+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":1989,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-03-30T13:26:49.612248+0000","last_change":"2024-03-30T13:26:19.523394+0000","last_active":"2024-03-30T13:26:49.612248+0000","last_peered":"2024-03-30T13:26:49.612248+0000","last_clean":"2024-03-30T13:26:49.612248+0000","last_became_active":"2024-03-30T13:26:16.582448+0000","last_became_peered":"2024-03-30T13:26:16.582448+0000","last_unstale":"2024-03-30T13:26:49.612248+0000","last_undegraded":"2024-03-30T13:26:49.612248+0000","last_fullsized":"2024-03-30T13:26:49.612248+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:26:19.523081+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:26:19.523081+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:39.559039+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":76134720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1628,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-03-30T13:26:49.402009+0000","last_change":"2024-03-30T13:26:48.759840+0000","last_active":"2024-03-30T13:26:49.402009+0000","last_peered":"2024-03-30T13:26:49.402009+0000","last_clean":"2024-03-30T13:26:49.402009+0000","last_became_active":"2024-03-30T13:24:29.442802+0000","last_became_peered":"2024-03-30T13:24:29.442802+0000","last_unstale":"2024-03-30T13:26:49.402009+0000","last_undegraded":"2024-03-30T13:26:49.402009+0000","last_fullsized":"2024-03-30T13:26:49.402009+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:26:48.759583+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:26:48.759583+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:58.679493+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1939,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-03-30T13:26:49.612078+0000","last_change":"2024-03-30T13:26:01.555623+0000","last_active":"2024-03-30T13:26:49.612078+0000","last_peered":"2024-03-30T13:26:49.612078+0000","last_clean":"2024-03-30T13:26:49.612078+0000","last_became_active":"2024-03-30T13:24:29.447202+0000","last_became_peered":"2024-03-30T13:24:29.447202+0000","last_unstale":"2024-03-30T13:26:49.612078+0000","last_undegraded":"2024-03-30T13:26:49.612078+0000","last_fullsized":"2024-03-30T13:26:49.612078+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:26:01.555372+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:26:01.555372+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:16.483568+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":2033,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-03-30T13:26:49.612152+0000","last_change":"2024-03-30T13:26:09.357933+0000","last_active":"2024-03-30T13:26:49.612152+0000","last_peered":"2024-03-30T13:26:49.612152+0000","last_clean":"2024-03-30T13:26:49.612152+0000","last_became_active":"2024-03-30T13:25:20.549465+0000","last_became_peered":"2024-03-30T13:25:20.549465+0000","last_unstale":"2024-03-30T13:26:49.612152+0000","last_undegraded":"2024-03-30T13:26:49.612152+0000","last_fullsized":"2024-03-30T13:26:49.612152+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:26:09.357648+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:26:09.357648+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:27.784454+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":786,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-03-30T13:26:49.612290+0000","last_change":"2024-03-30T13:26:35.032086+0000","last_active":"2024-03-30T13:26:49.612290+0000","last_peered":"2024-03-30T13:26:49.612290+0000","last_clean":"2024-03-30T13:26:49.612290+0000","last_became_active":"2024-03-30T13:25:20.549427+0000","last_became_peered":"2024-03-30T13:25:20.549427+0000","last_unstale":"2024-03-30T13:26:49.612290+0000","last_undegraded":"2024-03-30T13:26:49.612290+0000","last_fullsized":"2024-03-30T13:26:49.612290+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:26:35.031641+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:26:35.031641+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:42.633724+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":61,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-03-30T13:26:49.611377+0000","last_change":"2024-03-30T13:26:25.741558+0000","last_active":"2024-03-30T13:26:49.611377+0000","last_peered":"2024-03-30T13:26:49.611377+0000","last_clean":"2024-03-30T13:26:49.611377+0000","last_became_active":"2024-03-30T13:26:16.578484+0000","last_became_peered":"2024-03-30T13:26:16.578484+0000","last_unstale":"2024-03-30T13:26:49.611377+0000","last_undegraded":"2024-03-30T13:26:49.611377+0000","last_fullsized":"2024-03-30T13:26:49.611377+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:26:25.741212+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:26:25.741212+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:47.925369+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":801,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-03-30T13:26:49.612309+0000","last_change":"2024-03-30T13:26:09.903819+0000","last_active":"2024-03-30T13:26:49.612309+0000","last_peered":"2024-03-30T13:26:49.612309+0000","last_clean":"2024-03-30T13:26:49.612309+0000","last_became_active":"2024-03-30T13:25:20.556235+0000","last_became_peered":"2024-03-30T13:25:20.556235+0000","last_unstale":"2024-03-30T13:26:49.612309+0000","last_undegraded":"2024-03-30T13:26:49.612309+0000","last_fullsized":"2024-03-30T13:26:49.612309+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:26:09.903568+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_clean_scrub_stamp":"2024-03-30T13:26:09.903568+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:35.258779+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":23,"reported_epoch":859,"state":"active+clean","last_fresh":"2024-03-30T13:26:49.402435+0000","last_change":"2024-03-30T13:26:36.271554+0000","last_active":"2024-03-30T13:26:49.402435+0000","last_peered":"2024-03-30T13:26:49.402435+0000","last_clean":"2024-03-30T13:26:49.402435+0000","last_became_active":"2024-03-30T13:26:35.759232+0000","last_became_peered":"2024-03-30T13:26:35.759232+0000","last_unstale":"2024-03-30T13:26:49.402435+0000","last_undegraded":"2024-03-30T13:26:49.402435+0000","last_fullsized":"2024-03-30T13:26:49.402435+0000","mapping_epoch":855,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":856,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:25:58.642180+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:25:58.642180+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:21.134698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":5226911,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":281950470,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102486016,"data_stored":102486016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387774,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59608,"kb_used_data":22412,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94312232,"statfs":{"total":96636764160,"available":96575725568,"internally_reserved":0,"allocated":22949888,"data_stored":18861588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":816,"seq":3504693313558,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68964,"kb_used_data":27156,"kb_used_omap":12,"kb_used_meta":41779,"kb_avail":94302876,"statfs":{"total":96636764160,"available":96566145024,"internally_reserved":0,"allocated":27807744,"data_stored":23719444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12654,"internal_metadata":42782354},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":794,"seq":3410204033056,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":61340,"kb_used_data":20812,"kb_used_omap":15,"kb_used_meta":40496,"kb_avail":94310500,"statfs":{"total":96636764160,"available":96573952000,"internally_reserved":0,"allocated":21311488,"data_stored":17223188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15941,"internal_metadata":41468347},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509821,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56780,"kb_used_data":17276,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94315060,"statfs":{"total":96636764160,"available":96578621440,"internally_reserved":0,"allocated":17690624,"data_stored":13602324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509822,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56148,"kb_used_data":20804,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315692,"statfs":{"total":96636764160,"available":96579268608,"internally_reserved":0,"allocated":21303296,"data_stored":17214996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":837,"seq":3594887626763,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58388,"kb_used_data":17348,"kb_used_omap":17,"kb_used_meta":41006,"kb_avail":94313452,"statfs":{"total":96636764160,"available":96576974848,"internally_reserved":0,"allocated":17764352,"data_stored":13676052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17588,"internal_metadata":41990988},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570164,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56552,"kb_used_data":19804,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94315288,"statfs":{"total":96636764160,"available":96578854912,"internally_reserved":0,"allocated":20279296,"data_stored":16190996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":10211328,"data_stored":10211328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":13832192,"data_stored":13832192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:27:00.432 INFO:tasks.ceph.ceph_manager.ceph:all pgs are active or peered! 2024-03-30T13:27:00.432 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-30T13:27:00.433 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-03-30T13:27:00.433 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-30T13:27:00.433 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-30T13:27:00.434 INFO:tasks.ceph.osd.1:Started 2024-03-30T13:27:00.435 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-30T13:27:00.494 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:27:00.493+0000 7f63872097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:00.501 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:27:00.501+0000 7f63872097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:00.502 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:27:00.501+0000 7f63872097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:00.546 INFO:teuthology.orchestra.run.smithi067.stderr:no valid command found; 10 closest matches: 2024-03-30T13:27:00.546 INFO:teuthology.orchestra.run.smithi067.stderr:0 2024-03-30T13:27:00.546 INFO:teuthology.orchestra.run.smithi067.stderr:1 2024-03-30T13:27:00.546 INFO:teuthology.orchestra.run.smithi067.stderr:2 2024-03-30T13:27:00.546 INFO:teuthology.orchestra.run.smithi067.stderr:abort 2024-03-30T13:27:00.547 INFO:teuthology.orchestra.run.smithi067.stderr:assert 2024-03-30T13:27:00.547 INFO:teuthology.orchestra.run.smithi067.stderr:config diff 2024-03-30T13:27:00.547 INFO:teuthology.orchestra.run.smithi067.stderr:config diff get 2024-03-30T13:27:00.547 INFO:teuthology.orchestra.run.smithi067.stderr:config get 2024-03-30T13:27:00.547 INFO:teuthology.orchestra.run.smithi067.stderr:config help [] 2024-03-30T13:27:00.547 INFO:teuthology.orchestra.run.smithi067.stderr:config set ... 2024-03-30T13:27:00.547 INFO:teuthology.orchestra.run.smithi067.stderr:admin_socket: invalid command 2024-03-30T13:27:00.548 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-30T13:27:00.549 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2024-03-30T13:27:01.033 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:27:01.033+0000 7f63872097c0 -1 Falling back to public interface 2024-03-30T13:27:04.808 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:27:04.809+0000 7f63872097c0 -1 osd.1 854 log_to_monitors true 2024-03-30T13:27:05.488 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-03-30T13:27:05.489+0000 7f637efde640 -1 osd.1 854 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-30T13:27:05.550 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-03-30T13:27:05.685 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-30T13:27:05.685 INFO:teuthology.orchestra.run.smithi067.stdout: "ops": [], 2024-03-30T13:27:05.685 INFO:teuthology.orchestra.run.smithi067.stdout: "num_ops": 0 2024-03-30T13:27:05.685 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-30T13:27:05.699 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-30T13:27:05.700 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-03-30T13:27:05.700 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 1 2024-03-30T13:27:05.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:05.821+0000 7f2398624640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:05.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:05.837+0000 7f2398624640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:06.019 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-03-30T13:27:06.021+0000 7fbb38f08640 -1 mon.a@0(leader).osd e860 definitely_dead 0 2024-03-30T13:27:06.454 INFO:teuthology.orchestra.run.smithi067.stderr:marked in osd.1. 2024-03-30T13:27:06.483 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-03-30T13:27:06.484 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:27:06.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:06.597+0000 7f55688d2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:06.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:06.613+0000 7f55688d2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:07.467 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:27:07.495 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:27:07.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:07.613+0000 7f3ead7ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:07.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:07.633+0000 7f3ead7ac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:08.501 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:27:08.525 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:27:08.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:08.645+0000 7fdbcc393640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:08.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:08.665+0000 7fdbcc393640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:09.496 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:27:09.521 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:27:09.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:09.641+0000 7f993aafc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:09.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:09.657+0000 7f993aafc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:10.514 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:27:10.539 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:27:10.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:10.660+0000 7fb2ed97f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:10.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:10.676+0000 7fb2ed97f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:11.529 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:27:11.556 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:27:11.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:11.676+0000 7fe7c7168640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:11.692+0000 7fe7c7168640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:12.537 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:27:12.563 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:27:12.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:12.684+0000 7f0ebc656640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:12.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:12.700+0000 7f0ebc656640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:13.552 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:27:13.579 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:27:13.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:13.692+0000 7fce4afb6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:13.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:13.712+0000 7fce4afb6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:14.549 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:27:14.577 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:27:14.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:14.696+0000 7f0b05981640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:14.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:14.712+0000 7f0b05981640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:15.567 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:27:15.590 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:27:15.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:15.712+0000 7f5681610640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:15.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:15.728+0000 7f5681610640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:16.577 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:27:16.601 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:27:16.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:16.720+0000 7f88f3b5a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:16.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:16.736+0000 7f88f3b5a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:17.583 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:27:17.608 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:27:17.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:17.724+0000 7f6b6e35b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:17.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:17.740+0000 7f6b6e35b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:18.606 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:27:18.631 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:27:18.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:18.748+0000 7fa3a48aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:18.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:18.764+0000 7fa3a48aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:19.608 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:27:19.639 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:27:19.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:19.764+0000 7fcc2470d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:19.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:19.780+0000 7fcc2470d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:20.613 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:27:20.642 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:27:20.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:20.764+0000 7f42cbac5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:20.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:20.780+0000 7f42cbac5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:21.635 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:27:21.660 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:27:21.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:21.776+0000 7fcd2ad65640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:21.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:21.796+0000 7fcd2ad65640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:22.632 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:27:22.659 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:27:22.659 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:27:22.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:22.772+0000 7f289358b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:22.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:22.792+0000 7f289358b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:22.965 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:27:22.965 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:27:22.983 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1825,"stamp":"2024-03-30T13:27:22.635051+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":276723559,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":464096,"kb_used_data":154272,"kb_used_omap":77,"kb_used_meta":309618,"kb_avail":754510624,"statfs":{"total":773094113280,"available":772618878976,"internally_reserved":0,"allocated":157974528,"data_stored":124511969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":79062,"internal_metadata":317049642},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.032517"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1959,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-03-30T13:27:19.143340+0000","last_change":"2024-03-30T13:26:34.531020+0000","last_active":"2024-03-30T13:27:19.143340+0000","last_peered":"2024-03-30T13:27:19.143340+0000","last_clean":"2024-03-30T13:27:19.143340+0000","last_became_active":"2024-03-30T13:25:20.549084+0000","last_became_peered":"2024-03-30T13:25:20.549084+0000","last_unstale":"2024-03-30T13:27:19.143340+0000","last_undegraded":"2024-03-30T13:27:19.143340+0000","last_fullsized":"2024-03-30T13:27:19.143340+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:26:34.530597+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:25:12.543076+0000","last_clean_scrub_stamp":"2024-03-30T13:26:34.530597+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:35.601883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":13764735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1843,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-03-30T13:27:17.586849+0000","last_change":"2024-03-30T13:26:19.721841+0000","last_active":"2024-03-30T13:27:17.586849+0000","last_peered":"2024-03-30T13:27:17.586849+0000","last_clean":"2024-03-30T13:27:17.586849+0000","last_became_active":"2024-03-30T13:26:16.583003+0000","last_became_peered":"2024-03-30T13:26:16.583003+0000","last_unstale":"2024-03-30T13:27:17.586849+0000","last_undegraded":"2024-03-30T13:27:17.586849+0000","last_fullsized":"2024-03-30T13:27:17.586849+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:26:19.721595+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:26:19.721595+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:24.825128+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":115,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-30T13:27:20.185562+0000","last_change":"2024-03-30T13:27:07.484434+0000","last_active":"2024-03-30T13:27:20.185562+0000","last_peered":"2024-03-30T13:27:20.185562+0000","last_clean":"2024-03-30T13:27:20.185562+0000","last_became_active":"2024-03-30T13:27:07.482993+0000","last_became_peered":"2024-03-30T13:27:07.482993+0000","last_unstale":"2024-03-30T13:27:20.185562+0000","last_undegraded":"2024-03-30T13:27:20.185562+0000","last_fullsized":"2024-03-30T13:27:20.185562+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:26:47.672328+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:26:47.672328+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:02.109948+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":93,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-30T13:27:20.185444+0000","last_change":"2024-03-30T13:26:21.852195+0000","last_active":"2024-03-30T13:27:20.185444+0000","last_peered":"2024-03-30T13:27:20.185444+0000","last_clean":"2024-03-30T13:27:20.185444+0000","last_became_active":"2024-03-30T13:26:16.577250+0000","last_became_peered":"2024-03-30T13:26:16.577250+0000","last_unstale":"2024-03-30T13:27:20.185444+0000","last_undegraded":"2024-03-30T13:27:20.185444+0000","last_fullsized":"2024-03-30T13:27:20.185444+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:26:21.851948+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:26:21.851948+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:41.578957+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":769,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-03-30T13:27:19.142207+0000","last_change":"2024-03-30T13:27:01.991450+0000","last_active":"2024-03-30T13:27:19.142207+0000","last_peered":"2024-03-30T13:27:19.142207+0000","last_clean":"2024-03-30T13:27:19.142207+0000","last_became_active":"2024-03-30T13:25:20.556165+0000","last_became_peered":"2024-03-30T13:25:20.556165+0000","last_unstale":"2024-03-30T13:27:19.142207+0000","last_undegraded":"2024-03-30T13:27:19.142207+0000","last_fullsized":"2024-03-30T13:27:19.142207+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:01.991062+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:27:01.991062+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:06.741931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1900,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-30T13:27:20.622023+0000","last_change":"2024-03-30T13:26:25.441051+0000","last_active":"2024-03-30T13:27:20.622023+0000","last_peered":"2024-03-30T13:27:20.622023+0000","last_clean":"2024-03-30T13:27:20.622023+0000","last_became_active":"2024-03-30T13:25:20.551941+0000","last_became_peered":"2024-03-30T13:25:20.551941+0000","last_unstale":"2024-03-30T13:27:20.622023+0000","last_undegraded":"2024-03-30T13:27:20.622023+0000","last_fullsized":"2024-03-30T13:27:20.622023+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:26:25.440798+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:26:25.440798+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:36.431590+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11394981,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":108,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-30T13:27:20.185484+0000","last_change":"2024-03-30T13:27:12.597085+0000","last_active":"2024-03-30T13:27:20.185484+0000","last_peered":"2024-03-30T13:27:20.185484+0000","last_clean":"2024-03-30T13:27:20.185484+0000","last_became_active":"2024-03-30T13:26:16.583861+0000","last_became_peered":"2024-03-30T13:26:16.583861+0000","last_unstale":"2024-03-30T13:27:20.185484+0000","last_undegraded":"2024-03-30T13:27:20.185484+0000","last_fullsized":"2024-03-30T13:27:20.185484+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:27:12.596825+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:27:12.596825+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:25.628559+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1212,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-30T13:27:20.918612+0000","last_change":"2024-03-30T13:27:07.599735+0000","last_active":"2024-03-30T13:27:20.918612+0000","last_peered":"2024-03-30T13:27:20.918612+0000","last_clean":"2024-03-30T13:27:20.918612+0000","last_became_active":"2024-03-30T13:27:07.598881+0000","last_became_peered":"2024-03-30T13:27:07.598881+0000","last_unstale":"2024-03-30T13:27:20.918612+0000","last_undegraded":"2024-03-30T13:27:20.918612+0000","last_fullsized":"2024-03-30T13:27:20.918612+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:25:52.599859+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:25:52.599859+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":2021,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-30T13:27:20.621881+0000","last_change":"2024-03-30T13:26:19.523394+0000","last_active":"2024-03-30T13:27:20.621881+0000","last_peered":"2024-03-30T13:27:20.621881+0000","last_clean":"2024-03-30T13:27:20.621881+0000","last_became_active":"2024-03-30T13:26:16.582448+0000","last_became_peered":"2024-03-30T13:26:16.582448+0000","last_unstale":"2024-03-30T13:27:20.621881+0000","last_undegraded":"2024-03-30T13:27:20.621881+0000","last_fullsized":"2024-03-30T13:27:20.621881+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:26:19.523081+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:26:19.523081+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:39.559039+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":76134720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1653,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-03-30T13:27:17.586750+0000","last_change":"2024-03-30T13:26:48.759840+0000","last_active":"2024-03-30T13:27:17.586750+0000","last_peered":"2024-03-30T13:27:17.586750+0000","last_clean":"2024-03-30T13:27:17.586750+0000","last_became_active":"2024-03-30T13:24:29.442802+0000","last_became_peered":"2024-03-30T13:24:29.442802+0000","last_unstale":"2024-03-30T13:27:17.586750+0000","last_undegraded":"2024-03-30T13:27:17.586750+0000","last_fullsized":"2024-03-30T13:27:17.586750+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:26:48.759583+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:26:48.759583+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:58.679493+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1974,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-03-30T13:27:19.143152+0000","last_change":"2024-03-30T13:27:17.528668+0000","last_active":"2024-03-30T13:27:19.143152+0000","last_peered":"2024-03-30T13:27:19.143152+0000","last_clean":"2024-03-30T13:27:19.143152+0000","last_became_active":"2024-03-30T13:24:29.447202+0000","last_became_peered":"2024-03-30T13:24:29.447202+0000","last_unstale":"2024-03-30T13:27:19.143152+0000","last_undegraded":"2024-03-30T13:27:19.143152+0000","last_fullsized":"2024-03-30T13:27:19.143152+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:27:17.528414+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:27:17.528414+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:25.562324+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":2060,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-03-30T13:27:19.143185+0000","last_change":"2024-03-30T13:26:09.357933+0000","last_active":"2024-03-30T13:27:19.143185+0000","last_peered":"2024-03-30T13:27:19.143185+0000","last_clean":"2024-03-30T13:27:19.143185+0000","last_became_active":"2024-03-30T13:25:20.549465+0000","last_became_peered":"2024-03-30T13:25:20.549465+0000","last_unstale":"2024-03-30T13:27:19.143185+0000","last_undegraded":"2024-03-30T13:27:19.143185+0000","last_fullsized":"2024-03-30T13:27:19.143185+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:26:09.357648+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:26:09.357648+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:27.784454+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":813,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-03-30T13:27:19.141853+0000","last_change":"2024-03-30T13:26:35.032086+0000","last_active":"2024-03-30T13:27:19.141853+0000","last_peered":"2024-03-30T13:27:19.141853+0000","last_clean":"2024-03-30T13:27:19.141853+0000","last_became_active":"2024-03-30T13:25:20.549427+0000","last_became_peered":"2024-03-30T13:25:20.549427+0000","last_unstale":"2024-03-30T13:27:19.141853+0000","last_undegraded":"2024-03-30T13:27:19.141853+0000","last_fullsized":"2024-03-30T13:27:19.141853+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:26:35.031641+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:26:35.031641+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:42.633724+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":91,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-30T13:27:20.185473+0000","last_change":"2024-03-30T13:26:25.741558+0000","last_active":"2024-03-30T13:27:20.185473+0000","last_peered":"2024-03-30T13:27:20.185473+0000","last_clean":"2024-03-30T13:27:20.185473+0000","last_became_active":"2024-03-30T13:26:16.578484+0000","last_became_peered":"2024-03-30T13:26:16.578484+0000","last_unstale":"2024-03-30T13:27:20.185473+0000","last_undegraded":"2024-03-30T13:27:20.185473+0000","last_fullsized":"2024-03-30T13:27:20.185473+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:26:25.741212+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:26:25.741212+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:47.925369+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":828,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-03-30T13:27:19.141853+0000","last_change":"2024-03-30T13:26:09.903819+0000","last_active":"2024-03-30T13:27:19.141853+0000","last_peered":"2024-03-30T13:27:19.141853+0000","last_clean":"2024-03-30T13:27:19.141853+0000","last_became_active":"2024-03-30T13:25:20.556235+0000","last_became_peered":"2024-03-30T13:25:20.556235+0000","last_unstale":"2024-03-30T13:27:19.141853+0000","last_undegraded":"2024-03-30T13:27:19.141853+0000","last_fullsized":"2024-03-30T13:27:19.141853+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:26:09.903568+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_clean_scrub_stamp":"2024-03-30T13:26:09.903568+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:35.258779+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":1075,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-30T13:27:20.918681+0000","last_change":"2024-03-30T13:27:07.479482+0000","last_active":"2024-03-30T13:27:20.918681+0000","last_peered":"2024-03-30T13:27:20.918681+0000","last_clean":"2024-03-30T13:27:20.918681+0000","last_became_active":"2024-03-30T13:27:07.478858+0000","last_became_peered":"2024-03-30T13:27:07.478858+0000","last_unstale":"2024-03-30T13:27:20.918681+0000","last_undegraded":"2024-03-30T13:27:20.918681+0000","last_fullsized":"2024-03-30T13:27:20.918681+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:25:58.642180+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:25:58.642180+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:21.949985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":276723559,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387779,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59776,"kb_used_data":22580,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94312064,"statfs":{"total":96636764160,"available":96575553536,"internally_reserved":0,"allocated":23121920,"data_stored":18939944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":816,"seq":3504693313562,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69116,"kb_used_data":27312,"kb_used_omap":12,"kb_used_meta":41779,"kb_avail":94302724,"statfs":{"total":96636764160,"available":96565989376,"internally_reserved":0,"allocated":27967488,"data_stored":23792313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12654,"internal_metadata":42782354},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":794,"seq":3410204033060,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59548,"kb_used_data":19016,"kb_used_omap":15,"kb_used_meta":40496,"kb_avail":94312292,"statfs":{"total":96636764160,"available":96575787008,"internally_reserved":0,"allocated":19472384,"data_stored":15297209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15941,"internal_metadata":41468347},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509825,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53296,"kb_used_data":13796,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94318544,"statfs":{"total":96636764160,"available":96582189056,"internally_reserved":0,"allocated":14127104,"data_stored":9945128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":861,"seq":3697966841861,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50736,"kb_used_data":13084,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94321104,"statfs":{"total":96636764160,"available":96584810496,"internally_reserved":0,"allocated":13398016,"data_stored":9209239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12649,"internal_metadata":38522519},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509826,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56300,"kb_used_data":20960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315540,"statfs":{"total":96636764160,"available":96579112960,"internally_reserved":0,"allocated":21463040,"data_stored":17287865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":837,"seq":3594887626768,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58572,"kb_used_data":17528,"kb_used_omap":17,"kb_used_meta":41006,"kb_avail":94313268,"statfs":{"total":96636764160,"available":96576786432,"internally_reserved":0,"allocated":17948672,"data_stored":13759895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17588,"internal_metadata":41990988},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570169,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56752,"kb_used_data":19996,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94315088,"statfs":{"total":96636764160,"available":96578650112,"internally_reserved":0,"allocated":20475904,"data_stored":16280376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:27:22.985 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:27:23.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:23.100+0000 7f703b66c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:23.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:23.116+0000 7f703b66c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:23.291 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:27:23.292 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:27:23.310 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1825,"stamp":"2024-03-30T13:27:22.635051+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":276723559,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":464096,"kb_used_data":154272,"kb_used_omap":77,"kb_used_meta":309618,"kb_avail":754510624,"statfs":{"total":773094113280,"available":772618878976,"internally_reserved":0,"allocated":157974528,"data_stored":124511969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":79062,"internal_metadata":317049642},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.032517"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1959,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-03-30T13:27:19.143340+0000","last_change":"2024-03-30T13:26:34.531020+0000","last_active":"2024-03-30T13:27:19.143340+0000","last_peered":"2024-03-30T13:27:19.143340+0000","last_clean":"2024-03-30T13:27:19.143340+0000","last_became_active":"2024-03-30T13:25:20.549084+0000","last_became_peered":"2024-03-30T13:25:20.549084+0000","last_unstale":"2024-03-30T13:27:19.143340+0000","last_undegraded":"2024-03-30T13:27:19.143340+0000","last_fullsized":"2024-03-30T13:27:19.143340+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:26:34.530597+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:25:12.543076+0000","last_clean_scrub_stamp":"2024-03-30T13:26:34.530597+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:35.601883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":13764735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1843,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-03-30T13:27:17.586849+0000","last_change":"2024-03-30T13:26:19.721841+0000","last_active":"2024-03-30T13:27:17.586849+0000","last_peered":"2024-03-30T13:27:17.586849+0000","last_clean":"2024-03-30T13:27:17.586849+0000","last_became_active":"2024-03-30T13:26:16.583003+0000","last_became_peered":"2024-03-30T13:26:16.583003+0000","last_unstale":"2024-03-30T13:27:17.586849+0000","last_undegraded":"2024-03-30T13:27:17.586849+0000","last_fullsized":"2024-03-30T13:27:17.586849+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:26:19.721595+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:26:19.721595+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:24.825128+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":115,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-30T13:27:20.185562+0000","last_change":"2024-03-30T13:27:07.484434+0000","last_active":"2024-03-30T13:27:20.185562+0000","last_peered":"2024-03-30T13:27:20.185562+0000","last_clean":"2024-03-30T13:27:20.185562+0000","last_became_active":"2024-03-30T13:27:07.482993+0000","last_became_peered":"2024-03-30T13:27:07.482993+0000","last_unstale":"2024-03-30T13:27:20.185562+0000","last_undegraded":"2024-03-30T13:27:20.185562+0000","last_fullsized":"2024-03-30T13:27:20.185562+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:26:47.672328+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:26:47.672328+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:02.109948+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":93,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-30T13:27:20.185444+0000","last_change":"2024-03-30T13:26:21.852195+0000","last_active":"2024-03-30T13:27:20.185444+0000","last_peered":"2024-03-30T13:27:20.185444+0000","last_clean":"2024-03-30T13:27:20.185444+0000","last_became_active":"2024-03-30T13:26:16.577250+0000","last_became_peered":"2024-03-30T13:26:16.577250+0000","last_unstale":"2024-03-30T13:27:20.185444+0000","last_undegraded":"2024-03-30T13:27:20.185444+0000","last_fullsized":"2024-03-30T13:27:20.185444+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:26:21.851948+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:26:21.851948+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:41.578957+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":769,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-03-30T13:27:19.142207+0000","last_change":"2024-03-30T13:27:01.991450+0000","last_active":"2024-03-30T13:27:19.142207+0000","last_peered":"2024-03-30T13:27:19.142207+0000","last_clean":"2024-03-30T13:27:19.142207+0000","last_became_active":"2024-03-30T13:25:20.556165+0000","last_became_peered":"2024-03-30T13:25:20.556165+0000","last_unstale":"2024-03-30T13:27:19.142207+0000","last_undegraded":"2024-03-30T13:27:19.142207+0000","last_fullsized":"2024-03-30T13:27:19.142207+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:01.991062+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:27:01.991062+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:06.741931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1900,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-30T13:27:20.622023+0000","last_change":"2024-03-30T13:26:25.441051+0000","last_active":"2024-03-30T13:27:20.622023+0000","last_peered":"2024-03-30T13:27:20.622023+0000","last_clean":"2024-03-30T13:27:20.622023+0000","last_became_active":"2024-03-30T13:25:20.551941+0000","last_became_peered":"2024-03-30T13:25:20.551941+0000","last_unstale":"2024-03-30T13:27:20.622023+0000","last_undegraded":"2024-03-30T13:27:20.622023+0000","last_fullsized":"2024-03-30T13:27:20.622023+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:26:25.440798+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:26:25.440798+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:36.431590+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11394981,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":108,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-30T13:27:20.185484+0000","last_change":"2024-03-30T13:27:12.597085+0000","last_active":"2024-03-30T13:27:20.185484+0000","last_peered":"2024-03-30T13:27:20.185484+0000","last_clean":"2024-03-30T13:27:20.185484+0000","last_became_active":"2024-03-30T13:26:16.583861+0000","last_became_peered":"2024-03-30T13:26:16.583861+0000","last_unstale":"2024-03-30T13:27:20.185484+0000","last_undegraded":"2024-03-30T13:27:20.185484+0000","last_fullsized":"2024-03-30T13:27:20.185484+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:27:12.596825+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:27:12.596825+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:25.628559+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1212,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-30T13:27:20.918612+0000","last_change":"2024-03-30T13:27:07.599735+0000","last_active":"2024-03-30T13:27:20.918612+0000","last_peered":"2024-03-30T13:27:20.918612+0000","last_clean":"2024-03-30T13:27:20.918612+0000","last_became_active":"2024-03-30T13:27:07.598881+0000","last_became_peered":"2024-03-30T13:27:07.598881+0000","last_unstale":"2024-03-30T13:27:20.918612+0000","last_undegraded":"2024-03-30T13:27:20.918612+0000","last_fullsized":"2024-03-30T13:27:20.918612+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:25:52.599859+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:25:52.599859+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":2021,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-30T13:27:20.621881+0000","last_change":"2024-03-30T13:26:19.523394+0000","last_active":"2024-03-30T13:27:20.621881+0000","last_peered":"2024-03-30T13:27:20.621881+0000","last_clean":"2024-03-30T13:27:20.621881+0000","last_became_active":"2024-03-30T13:26:16.582448+0000","last_became_peered":"2024-03-30T13:26:16.582448+0000","last_unstale":"2024-03-30T13:27:20.621881+0000","last_undegraded":"2024-03-30T13:27:20.621881+0000","last_fullsized":"2024-03-30T13:27:20.621881+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:26:19.523081+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:26:19.523081+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:39.559039+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":76134720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1653,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-03-30T13:27:17.586750+0000","last_change":"2024-03-30T13:26:48.759840+0000","last_active":"2024-03-30T13:27:17.586750+0000","last_peered":"2024-03-30T13:27:17.586750+0000","last_clean":"2024-03-30T13:27:17.586750+0000","last_became_active":"2024-03-30T13:24:29.442802+0000","last_became_peered":"2024-03-30T13:24:29.442802+0000","last_unstale":"2024-03-30T13:27:17.586750+0000","last_undegraded":"2024-03-30T13:27:17.586750+0000","last_fullsized":"2024-03-30T13:27:17.586750+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:26:48.759583+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:26:48.759583+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:58.679493+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":1974,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-03-30T13:27:19.143152+0000","last_change":"2024-03-30T13:27:17.528668+0000","last_active":"2024-03-30T13:27:19.143152+0000","last_peered":"2024-03-30T13:27:19.143152+0000","last_clean":"2024-03-30T13:27:19.143152+0000","last_became_active":"2024-03-30T13:24:29.447202+0000","last_became_peered":"2024-03-30T13:24:29.447202+0000","last_unstale":"2024-03-30T13:27:19.143152+0000","last_undegraded":"2024-03-30T13:27:19.143152+0000","last_fullsized":"2024-03-30T13:27:19.143152+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:27:17.528414+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:27:17.528414+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:25.562324+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":2060,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-03-30T13:27:19.143185+0000","last_change":"2024-03-30T13:26:09.357933+0000","last_active":"2024-03-30T13:27:19.143185+0000","last_peered":"2024-03-30T13:27:19.143185+0000","last_clean":"2024-03-30T13:27:19.143185+0000","last_became_active":"2024-03-30T13:25:20.549465+0000","last_became_peered":"2024-03-30T13:25:20.549465+0000","last_unstale":"2024-03-30T13:27:19.143185+0000","last_undegraded":"2024-03-30T13:27:19.143185+0000","last_fullsized":"2024-03-30T13:27:19.143185+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:26:09.357648+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:26:09.357648+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:27.784454+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":813,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-03-30T13:27:19.141853+0000","last_change":"2024-03-30T13:26:35.032086+0000","last_active":"2024-03-30T13:27:19.141853+0000","last_peered":"2024-03-30T13:27:19.141853+0000","last_clean":"2024-03-30T13:27:19.141853+0000","last_became_active":"2024-03-30T13:25:20.549427+0000","last_became_peered":"2024-03-30T13:25:20.549427+0000","last_unstale":"2024-03-30T13:27:19.141853+0000","last_undegraded":"2024-03-30T13:27:19.141853+0000","last_fullsized":"2024-03-30T13:27:19.141853+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:26:35.031641+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:26:35.031641+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:42.633724+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":91,"reported_epoch":874,"state":"active+clean","last_fresh":"2024-03-30T13:27:20.185473+0000","last_change":"2024-03-30T13:26:25.741558+0000","last_active":"2024-03-30T13:27:20.185473+0000","last_peered":"2024-03-30T13:27:20.185473+0000","last_clean":"2024-03-30T13:27:20.185473+0000","last_became_active":"2024-03-30T13:26:16.578484+0000","last_became_peered":"2024-03-30T13:26:16.578484+0000","last_unstale":"2024-03-30T13:27:20.185473+0000","last_undegraded":"2024-03-30T13:27:20.185473+0000","last_fullsized":"2024-03-30T13:27:20.185473+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:26:25.741212+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:26:25.741212+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:47.925369+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":828,"reported_epoch":873,"state":"active+clean","last_fresh":"2024-03-30T13:27:19.141853+0000","last_change":"2024-03-30T13:26:09.903819+0000","last_active":"2024-03-30T13:27:19.141853+0000","last_peered":"2024-03-30T13:27:19.141853+0000","last_clean":"2024-03-30T13:27:19.141853+0000","last_became_active":"2024-03-30T13:25:20.556235+0000","last_became_peered":"2024-03-30T13:25:20.556235+0000","last_unstale":"2024-03-30T13:27:19.141853+0000","last_undegraded":"2024-03-30T13:27:19.141853+0000","last_fullsized":"2024-03-30T13:27:19.141853+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:26:09.903568+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_clean_scrub_stamp":"2024-03-30T13:26:09.903568+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:35.258779+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":1075,"reported_epoch":875,"state":"active+clean","last_fresh":"2024-03-30T13:27:20.918681+0000","last_change":"2024-03-30T13:27:07.479482+0000","last_active":"2024-03-30T13:27:20.918681+0000","last_peered":"2024-03-30T13:27:20.918681+0000","last_clean":"2024-03-30T13:27:20.918681+0000","last_became_active":"2024-03-30T13:27:07.478858+0000","last_became_peered":"2024-03-30T13:27:07.478858+0000","last_unstale":"2024-03-30T13:27:20.918681+0000","last_undegraded":"2024-03-30T13:27:20.918681+0000","last_fullsized":"2024-03-30T13:27:20.918681+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:25:58.642180+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:25:58.642180+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:21.949985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":276723559,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387779,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59776,"kb_used_data":22580,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94312064,"statfs":{"total":96636764160,"available":96575553536,"internally_reserved":0,"allocated":23121920,"data_stored":18939944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":816,"seq":3504693313562,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69116,"kb_used_data":27312,"kb_used_omap":12,"kb_used_meta":41779,"kb_avail":94302724,"statfs":{"total":96636764160,"available":96565989376,"internally_reserved":0,"allocated":27967488,"data_stored":23792313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12654,"internal_metadata":42782354},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":794,"seq":3410204033060,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59548,"kb_used_data":19016,"kb_used_omap":15,"kb_used_meta":40496,"kb_avail":94312292,"statfs":{"total":96636764160,"available":96575787008,"internally_reserved":0,"allocated":19472384,"data_stored":15297209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15941,"internal_metadata":41468347},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509825,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53296,"kb_used_data":13796,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94318544,"statfs":{"total":96636764160,"available":96582189056,"internally_reserved":0,"allocated":14127104,"data_stored":9945128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":861,"seq":3697966841861,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50736,"kb_used_data":13084,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94321104,"statfs":{"total":96636764160,"available":96584810496,"internally_reserved":0,"allocated":13398016,"data_stored":9209239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12649,"internal_metadata":38522519},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509826,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56300,"kb_used_data":20960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315540,"statfs":{"total":96636764160,"available":96579112960,"internally_reserved":0,"allocated":21463040,"data_stored":17287865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":837,"seq":3594887626768,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58572,"kb_used_data":17528,"kb_used_omap":17,"kb_used_meta":41006,"kb_avail":94313268,"statfs":{"total":96636764160,"available":96576786432,"internally_reserved":0,"allocated":17948672,"data_stored":13759895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17588,"internal_metadata":41990988},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570169,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56752,"kb_used_data":19996,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94315088,"statfs":{"total":96636764160,"available":96578650112,"internally_reserved":0,"allocated":20475904,"data_stored":16280376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:27:23.311 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:27:23.311 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:27:23.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:23.424+0000 7fa5a8cf2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:23.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:23.440+0000 7fa5a8cf2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:23.654 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:27:23.675 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:27:23.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:23.792+0000 7efcfa2a1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:23.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:23.808+0000 7efcfa2a1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:24.656 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:27:24.681 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:27:24.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:24.800+0000 7f6d4497a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:24.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:24.816+0000 7f6d4497a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:25.668 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:27:25.696 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:27:25.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:25.812+0000 7fbcaffe7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:25.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:25.828+0000 7fbcaffe7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:26.686 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:27:26.711 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:27:26.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:26.832+0000 7fe62f4ea640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:26.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:26.848+0000 7fe62f4ea640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:27.691 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:27:27.721 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:27:27.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:27.840+0000 7ff8872a1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:27.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:27.856+0000 7ff8872a1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:28.708 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:27:28.732 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:27:28.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:28.852+0000 7fa2abe45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:28.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:28.868+0000 7fa2abe45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:29.718 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:27:29.742 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:27:29.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:29.856+0000 7f392f8ec640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:29.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:29.872+0000 7f392f8ec640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:30.731 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:27:30.760 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:27:30.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:30.880+0000 7fece2369640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:30.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:30.896+0000 7fece2369640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:31.747 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:27:31.773 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:27:31.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:31.892+0000 7fd4a20a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:31.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:31.908+0000 7fd4a20a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:32.755 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:27:32.779 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:27:32.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:32.900+0000 7f81c7587640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:32.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:32.916+0000 7f81c7587640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:33.771 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:27:33.794 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:27:33.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:33.912+0000 7ff85cf3c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:33.928+0000 7ff85cf3c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:34.768 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:27:34.799 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:27:34.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:34.924+0000 7f6b28720640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:34.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:34.940+0000 7f6b28720640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:35.794 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:27:35.818 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:27:35.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:35.940+0000 7f66ff792640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:35.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:35.956+0000 7f66ff792640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:36.789 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:27:36.815 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:27:36.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:36.932+0000 7fe8a15ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:36.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:36.948+0000 7fe8a15ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:37.800 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:27:37.826 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:27:37.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:37.944+0000 7f1c6b3aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:37.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:37.960+0000 7f1c6b3aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:38.823 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:27:38.847 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-30T13:27:38.847 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:27:38.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:38.964+0000 7f118b3e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:38.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:38.980+0000 7f118b3e4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:39.155 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:27:39.156 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:27:39.174 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1849,"stamp":"2024-03-30T13:27:38.820246+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":276723559,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":465576,"kb_used_data":155748,"kb_used_omap":77,"kb_used_meta":309618,"kb_avail":754509144,"statfs":{"total":773094113280,"available":772617363456,"internally_reserved":0,"allocated":159485952,"data_stored":125187220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":79062,"internal_metadata":317049642},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.048686"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1987,"reported_epoch":888,"state":"active+clean","last_fresh":"2024-03-30T13:27:33.771160+0000","last_change":"2024-03-30T13:26:34.531020+0000","last_active":"2024-03-30T13:27:33.771160+0000","last_peered":"2024-03-30T13:27:33.771160+0000","last_clean":"2024-03-30T13:27:33.771160+0000","last_became_active":"2024-03-30T13:25:20.549084+0000","last_became_peered":"2024-03-30T13:25:20.549084+0000","last_unstale":"2024-03-30T13:27:33.771160+0000","last_undegraded":"2024-03-30T13:27:33.771160+0000","last_fullsized":"2024-03-30T13:27:33.771160+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:26:34.530597+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:25:12.543076+0000","last_clean_scrub_stamp":"2024-03-30T13:26:34.530597+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:35.601883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":13764735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1888,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-30T13:27:37.807650+0000","last_change":"2024-03-30T13:27:26.798457+0000","last_active":"2024-03-30T13:27:37.807650+0000","last_peered":"2024-03-30T13:27:37.807650+0000","last_clean":"2024-03-30T13:27:37.807650+0000","last_became_active":"2024-03-30T13:26:16.583003+0000","last_became_peered":"2024-03-30T13:26:16.583003+0000","last_unstale":"2024-03-30T13:27:37.807650+0000","last_undegraded":"2024-03-30T13:27:37.807650+0000","last_fullsized":"2024-03-30T13:27:37.807650+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:26.798199+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:27:26.798199+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:54.470648+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":146,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-30T13:27:35.791808+0000","last_change":"2024-03-30T13:27:07.484434+0000","last_active":"2024-03-30T13:27:35.791808+0000","last_peered":"2024-03-30T13:27:35.791808+0000","last_clean":"2024-03-30T13:27:35.791808+0000","last_became_active":"2024-03-30T13:27:07.482993+0000","last_became_peered":"2024-03-30T13:27:07.482993+0000","last_unstale":"2024-03-30T13:27:35.791808+0000","last_undegraded":"2024-03-30T13:27:35.791808+0000","last_fullsized":"2024-03-30T13:27:35.791808+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:26:47.672328+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:26:47.672328+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:02.109948+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":124,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-30T13:27:35.791725+0000","last_change":"2024-03-30T13:26:21.852195+0000","last_active":"2024-03-30T13:27:35.791725+0000","last_peered":"2024-03-30T13:27:35.791725+0000","last_clean":"2024-03-30T13:27:35.791725+0000","last_became_active":"2024-03-30T13:26:16.577250+0000","last_became_peered":"2024-03-30T13:26:16.577250+0000","last_unstale":"2024-03-30T13:27:35.791725+0000","last_undegraded":"2024-03-30T13:27:35.791725+0000","last_fullsized":"2024-03-30T13:27:35.791725+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:26:21.851948+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:26:21.851948+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:41.578957+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":797,"reported_epoch":888,"state":"active+clean","last_fresh":"2024-03-30T13:27:34.420544+0000","last_change":"2024-03-30T13:27:01.991450+0000","last_active":"2024-03-30T13:27:34.420544+0000","last_peered":"2024-03-30T13:27:34.420544+0000","last_clean":"2024-03-30T13:27:34.420544+0000","last_became_active":"2024-03-30T13:25:20.556165+0000","last_became_peered":"2024-03-30T13:25:20.556165+0000","last_unstale":"2024-03-30T13:27:34.420544+0000","last_undegraded":"2024-03-30T13:27:34.420544+0000","last_fullsized":"2024-03-30T13:27:34.420544+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:01.991062+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:27:01.991062+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:06.741931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1929,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-30T13:27:35.794399+0000","last_change":"2024-03-30T13:26:25.441051+0000","last_active":"2024-03-30T13:27:35.794399+0000","last_peered":"2024-03-30T13:27:35.794399+0000","last_clean":"2024-03-30T13:27:35.794399+0000","last_became_active":"2024-03-30T13:25:20.551941+0000","last_became_peered":"2024-03-30T13:25:20.551941+0000","last_unstale":"2024-03-30T13:27:35.794399+0000","last_undegraded":"2024-03-30T13:27:35.794399+0000","last_fullsized":"2024-03-30T13:27:35.794399+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:26:25.440798+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:26:25.440798+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:36.431590+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11394981,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":139,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-30T13:27:35.791763+0000","last_change":"2024-03-30T13:27:12.597085+0000","last_active":"2024-03-30T13:27:35.791763+0000","last_peered":"2024-03-30T13:27:35.791763+0000","last_clean":"2024-03-30T13:27:35.791763+0000","last_became_active":"2024-03-30T13:26:16.583861+0000","last_became_peered":"2024-03-30T13:26:16.583861+0000","last_unstale":"2024-03-30T13:27:35.791763+0000","last_undegraded":"2024-03-30T13:27:35.791763+0000","last_fullsized":"2024-03-30T13:27:35.791763+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:27:12.596825+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:27:12.596825+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:25.628559+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1245,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-30T13:27:36.045793+0000","last_change":"2024-03-30T13:27:22.661584+0000","last_active":"2024-03-30T13:27:36.045793+0000","last_peered":"2024-03-30T13:27:36.045793+0000","last_clean":"2024-03-30T13:27:36.045793+0000","last_became_active":"2024-03-30T13:27:07.598881+0000","last_became_peered":"2024-03-30T13:27:07.598881+0000","last_unstale":"2024-03-30T13:27:36.045793+0000","last_undegraded":"2024-03-30T13:27:36.045793+0000","last_fullsized":"2024-03-30T13:27:36.045793+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:27:22.661428+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:27:22.661428+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:51.172824+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":2050,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-30T13:27:35.794600+0000","last_change":"2024-03-30T13:26:19.523394+0000","last_active":"2024-03-30T13:27:35.794600+0000","last_peered":"2024-03-30T13:27:35.794600+0000","last_clean":"2024-03-30T13:27:35.794600+0000","last_became_active":"2024-03-30T13:26:16.582448+0000","last_became_peered":"2024-03-30T13:26:16.582448+0000","last_unstale":"2024-03-30T13:27:35.794600+0000","last_undegraded":"2024-03-30T13:27:35.794600+0000","last_fullsized":"2024-03-30T13:27:35.794600+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:26:19.523081+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:26:19.523081+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:39.559039+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":76134720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1690,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-30T13:27:37.807577+0000","last_change":"2024-03-30T13:26:48.759840+0000","last_active":"2024-03-30T13:27:37.807577+0000","last_peered":"2024-03-30T13:27:37.807577+0000","last_clean":"2024-03-30T13:27:37.807577+0000","last_became_active":"2024-03-30T13:24:29.442802+0000","last_became_peered":"2024-03-30T13:24:29.442802+0000","last_unstale":"2024-03-30T13:27:37.807577+0000","last_undegraded":"2024-03-30T13:27:37.807577+0000","last_fullsized":"2024-03-30T13:27:37.807577+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:26:48.759583+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:26:48.759583+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:58.679493+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":2002,"reported_epoch":888,"state":"active+clean","last_fresh":"2024-03-30T13:27:33.771072+0000","last_change":"2024-03-30T13:27:17.528668+0000","last_active":"2024-03-30T13:27:33.771072+0000","last_peered":"2024-03-30T13:27:33.771072+0000","last_clean":"2024-03-30T13:27:33.771072+0000","last_became_active":"2024-03-30T13:24:29.447202+0000","last_became_peered":"2024-03-30T13:24:29.447202+0000","last_unstale":"2024-03-30T13:27:33.771072+0000","last_undegraded":"2024-03-30T13:27:33.771072+0000","last_fullsized":"2024-03-30T13:27:33.771072+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:27:17.528414+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:27:17.528414+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:25.562324+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":2096,"reported_epoch":888,"state":"active+clean","last_fresh":"2024-03-30T13:27:33.771146+0000","last_change":"2024-03-30T13:27:28.612011+0000","last_active":"2024-03-30T13:27:33.771146+0000","last_peered":"2024-03-30T13:27:33.771146+0000","last_clean":"2024-03-30T13:27:33.771146+0000","last_became_active":"2024-03-30T13:25:20.549465+0000","last_became_peered":"2024-03-30T13:25:20.549465+0000","last_unstale":"2024-03-30T13:27:33.771146+0000","last_undegraded":"2024-03-30T13:27:33.771146+0000","last_fullsized":"2024-03-30T13:27:33.771146+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:27:28.611732+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:27:28.611732+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:50.237192+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":842,"reported_epoch":888,"state":"active+clean","last_fresh":"2024-03-30T13:27:34.420547+0000","last_change":"2024-03-30T13:26:35.032086+0000","last_active":"2024-03-30T13:27:34.420547+0000","last_peered":"2024-03-30T13:27:34.420547+0000","last_clean":"2024-03-30T13:27:34.420547+0000","last_became_active":"2024-03-30T13:25:20.549427+0000","last_became_peered":"2024-03-30T13:25:20.549427+0000","last_unstale":"2024-03-30T13:27:34.420547+0000","last_undegraded":"2024-03-30T13:27:34.420547+0000","last_fullsized":"2024-03-30T13:27:34.420547+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:26:35.031641+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:26:35.031641+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:42.633724+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":122,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-30T13:27:35.791735+0000","last_change":"2024-03-30T13:26:25.741558+0000","last_active":"2024-03-30T13:27:35.791735+0000","last_peered":"2024-03-30T13:27:35.791735+0000","last_clean":"2024-03-30T13:27:35.791735+0000","last_became_active":"2024-03-30T13:26:16.578484+0000","last_became_peered":"2024-03-30T13:26:16.578484+0000","last_unstale":"2024-03-30T13:27:35.791735+0000","last_undegraded":"2024-03-30T13:27:35.791735+0000","last_fullsized":"2024-03-30T13:27:35.791735+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:26:25.741212+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:26:25.741212+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:47.925369+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":857,"reported_epoch":888,"state":"active+clean","last_fresh":"2024-03-30T13:27:34.420588+0000","last_change":"2024-03-30T13:26:09.903819+0000","last_active":"2024-03-30T13:27:34.420588+0000","last_peered":"2024-03-30T13:27:34.420588+0000","last_clean":"2024-03-30T13:27:34.420588+0000","last_became_active":"2024-03-30T13:25:20.556235+0000","last_became_peered":"2024-03-30T13:25:20.556235+0000","last_unstale":"2024-03-30T13:27:34.420588+0000","last_undegraded":"2024-03-30T13:27:34.420588+0000","last_fullsized":"2024-03-30T13:27:34.420588+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:26:09.903568+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_clean_scrub_stamp":"2024-03-30T13:26:09.903568+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:35.258779+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":1108,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-30T13:27:36.045790+0000","last_change":"2024-03-30T13:27:30.772468+0000","last_active":"2024-03-30T13:27:36.045790+0000","last_peered":"2024-03-30T13:27:36.045790+0000","last_clean":"2024-03-30T13:27:36.045790+0000","last_became_active":"2024-03-30T13:27:07.478858+0000","last_became_peered":"2024-03-30T13:27:07.478858+0000","last_unstale":"2024-03-30T13:27:36.045790+0000","last_undegraded":"2024-03-30T13:27:36.045790+0000","last_fullsized":"2024-03-30T13:27:36.045790+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:27:30.772331+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:27:30.772331+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:38.830587+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":276723559,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387782,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59964,"kb_used_data":22760,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94311876,"statfs":{"total":96636764160,"available":96575361024,"internally_reserved":0,"allocated":23306240,"data_stored":19022299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":816,"seq":3504693313565,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69288,"kb_used_data":27480,"kb_used_omap":12,"kb_used_meta":41779,"kb_avail":94302552,"statfs":{"total":96636764160,"available":96565813248,"internally_reserved":0,"allocated":28139520,"data_stored":23869181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12654,"internal_metadata":42782354},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":794,"seq":3410204033064,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59784,"kb_used_data":19256,"kb_used_omap":15,"kb_used_meta":40496,"kb_avail":94312056,"statfs":{"total":96636764160,"available":96575545344,"internally_reserved":0,"allocated":19718144,"data_stored":15406999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15941,"internal_metadata":41468347},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509828,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53476,"kb_used_data":13976,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94318364,"statfs":{"total":96636764160,"available":96582004736,"internally_reserved":0,"allocated":14311424,"data_stored":10027483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":861,"seq":3697966841865,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50900,"kb_used_data":13252,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94320940,"statfs":{"total":96636764160,"available":96584642560,"internally_reserved":0,"allocated":13570048,"data_stored":9286107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12649,"internal_metadata":38522519},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509829,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56480,"kb_used_data":21140,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315360,"statfs":{"total":96636764160,"available":96578928640,"internally_reserved":0,"allocated":21647360,"data_stored":17370220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":837,"seq":3594887626771,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58748,"kb_used_data":17708,"kb_used_omap":17,"kb_used_meta":41006,"kb_avail":94313092,"statfs":{"total":96636764160,"available":96576606208,"internally_reserved":0,"allocated":18132992,"data_stored":13842250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17588,"internal_metadata":41990988},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570172,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56936,"kb_used_data":20176,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94314904,"statfs":{"total":96636764160,"available":96578461696,"internally_reserved":0,"allocated":20660224,"data_stored":16362681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:27:39.175 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:27:39.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:39.296+0000 7f9d7b156640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:39.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:39.308+0000 7f9d7b156640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:39.483 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:27:39.484 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:27:39.501 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1849,"stamp":"2024-03-30T13:27:38.820246+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":276723559,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":465576,"kb_used_data":155748,"kb_used_omap":77,"kb_used_meta":309618,"kb_avail":754509144,"statfs":{"total":773094113280,"available":772617363456,"internally_reserved":0,"allocated":159485952,"data_stored":125187220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":79062,"internal_metadata":317049642},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.048686"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":1987,"reported_epoch":888,"state":"active+clean","last_fresh":"2024-03-30T13:27:33.771160+0000","last_change":"2024-03-30T13:26:34.531020+0000","last_active":"2024-03-30T13:27:33.771160+0000","last_peered":"2024-03-30T13:27:33.771160+0000","last_clean":"2024-03-30T13:27:33.771160+0000","last_became_active":"2024-03-30T13:25:20.549084+0000","last_became_peered":"2024-03-30T13:25:20.549084+0000","last_unstale":"2024-03-30T13:27:33.771160+0000","last_undegraded":"2024-03-30T13:27:33.771160+0000","last_fullsized":"2024-03-30T13:27:33.771160+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:26:34.530597+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:25:12.543076+0000","last_clean_scrub_stamp":"2024-03-30T13:26:34.530597+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:35.601883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":13764735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1888,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-30T13:27:37.807650+0000","last_change":"2024-03-30T13:27:26.798457+0000","last_active":"2024-03-30T13:27:37.807650+0000","last_peered":"2024-03-30T13:27:37.807650+0000","last_clean":"2024-03-30T13:27:37.807650+0000","last_became_active":"2024-03-30T13:26:16.583003+0000","last_became_peered":"2024-03-30T13:26:16.583003+0000","last_unstale":"2024-03-30T13:27:37.807650+0000","last_undegraded":"2024-03-30T13:27:37.807650+0000","last_fullsized":"2024-03-30T13:27:37.807650+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:26.798199+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:27:26.798199+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:54.470648+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":146,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-30T13:27:35.791808+0000","last_change":"2024-03-30T13:27:07.484434+0000","last_active":"2024-03-30T13:27:35.791808+0000","last_peered":"2024-03-30T13:27:35.791808+0000","last_clean":"2024-03-30T13:27:35.791808+0000","last_became_active":"2024-03-30T13:27:07.482993+0000","last_became_peered":"2024-03-30T13:27:07.482993+0000","last_unstale":"2024-03-30T13:27:35.791808+0000","last_undegraded":"2024-03-30T13:27:35.791808+0000","last_fullsized":"2024-03-30T13:27:35.791808+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:26:47.672328+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:26:47.672328+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:02.109948+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":124,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-30T13:27:35.791725+0000","last_change":"2024-03-30T13:26:21.852195+0000","last_active":"2024-03-30T13:27:35.791725+0000","last_peered":"2024-03-30T13:27:35.791725+0000","last_clean":"2024-03-30T13:27:35.791725+0000","last_became_active":"2024-03-30T13:26:16.577250+0000","last_became_peered":"2024-03-30T13:26:16.577250+0000","last_unstale":"2024-03-30T13:27:35.791725+0000","last_undegraded":"2024-03-30T13:27:35.791725+0000","last_fullsized":"2024-03-30T13:27:35.791725+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:26:21.851948+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:26:21.851948+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:41.578957+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":797,"reported_epoch":888,"state":"active+clean","last_fresh":"2024-03-30T13:27:34.420544+0000","last_change":"2024-03-30T13:27:01.991450+0000","last_active":"2024-03-30T13:27:34.420544+0000","last_peered":"2024-03-30T13:27:34.420544+0000","last_clean":"2024-03-30T13:27:34.420544+0000","last_became_active":"2024-03-30T13:25:20.556165+0000","last_became_peered":"2024-03-30T13:25:20.556165+0000","last_unstale":"2024-03-30T13:27:34.420544+0000","last_undegraded":"2024-03-30T13:27:34.420544+0000","last_fullsized":"2024-03-30T13:27:34.420544+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:01.991062+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:27:01.991062+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:06.741931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1929,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-30T13:27:35.794399+0000","last_change":"2024-03-30T13:26:25.441051+0000","last_active":"2024-03-30T13:27:35.794399+0000","last_peered":"2024-03-30T13:27:35.794399+0000","last_clean":"2024-03-30T13:27:35.794399+0000","last_became_active":"2024-03-30T13:25:20.551941+0000","last_became_peered":"2024-03-30T13:25:20.551941+0000","last_unstale":"2024-03-30T13:27:35.794399+0000","last_undegraded":"2024-03-30T13:27:35.794399+0000","last_fullsized":"2024-03-30T13:27:35.794399+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:26:25.440798+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:26:25.440798+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:36.431590+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11394981,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":139,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-30T13:27:35.791763+0000","last_change":"2024-03-30T13:27:12.597085+0000","last_active":"2024-03-30T13:27:35.791763+0000","last_peered":"2024-03-30T13:27:35.791763+0000","last_clean":"2024-03-30T13:27:35.791763+0000","last_became_active":"2024-03-30T13:26:16.583861+0000","last_became_peered":"2024-03-30T13:26:16.583861+0000","last_unstale":"2024-03-30T13:27:35.791763+0000","last_undegraded":"2024-03-30T13:27:35.791763+0000","last_fullsized":"2024-03-30T13:27:35.791763+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:27:12.596825+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:27:12.596825+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:25.628559+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1245,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-30T13:27:36.045793+0000","last_change":"2024-03-30T13:27:22.661584+0000","last_active":"2024-03-30T13:27:36.045793+0000","last_peered":"2024-03-30T13:27:36.045793+0000","last_clean":"2024-03-30T13:27:36.045793+0000","last_became_active":"2024-03-30T13:27:07.598881+0000","last_became_peered":"2024-03-30T13:27:07.598881+0000","last_unstale":"2024-03-30T13:27:36.045793+0000","last_undegraded":"2024-03-30T13:27:36.045793+0000","last_fullsized":"2024-03-30T13:27:36.045793+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:27:22.661428+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:27:22.661428+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:51.172824+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":2050,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-30T13:27:35.794600+0000","last_change":"2024-03-30T13:26:19.523394+0000","last_active":"2024-03-30T13:27:35.794600+0000","last_peered":"2024-03-30T13:27:35.794600+0000","last_clean":"2024-03-30T13:27:35.794600+0000","last_became_active":"2024-03-30T13:26:16.582448+0000","last_became_peered":"2024-03-30T13:26:16.582448+0000","last_unstale":"2024-03-30T13:27:35.794600+0000","last_undegraded":"2024-03-30T13:27:35.794600+0000","last_fullsized":"2024-03-30T13:27:35.794600+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:26:19.523081+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:26:19.523081+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:39.559039+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":76134720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1690,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-03-30T13:27:37.807577+0000","last_change":"2024-03-30T13:26:48.759840+0000","last_active":"2024-03-30T13:27:37.807577+0000","last_peered":"2024-03-30T13:27:37.807577+0000","last_clean":"2024-03-30T13:27:37.807577+0000","last_became_active":"2024-03-30T13:24:29.442802+0000","last_became_peered":"2024-03-30T13:24:29.442802+0000","last_unstale":"2024-03-30T13:27:37.807577+0000","last_undegraded":"2024-03-30T13:27:37.807577+0000","last_fullsized":"2024-03-30T13:27:37.807577+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:26:48.759583+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:26:48.759583+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:58.679493+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":2002,"reported_epoch":888,"state":"active+clean","last_fresh":"2024-03-30T13:27:33.771072+0000","last_change":"2024-03-30T13:27:17.528668+0000","last_active":"2024-03-30T13:27:33.771072+0000","last_peered":"2024-03-30T13:27:33.771072+0000","last_clean":"2024-03-30T13:27:33.771072+0000","last_became_active":"2024-03-30T13:24:29.447202+0000","last_became_peered":"2024-03-30T13:24:29.447202+0000","last_unstale":"2024-03-30T13:27:33.771072+0000","last_undegraded":"2024-03-30T13:27:33.771072+0000","last_fullsized":"2024-03-30T13:27:33.771072+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:27:17.528414+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:27:17.528414+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:25.562324+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":2096,"reported_epoch":888,"state":"active+clean","last_fresh":"2024-03-30T13:27:33.771146+0000","last_change":"2024-03-30T13:27:28.612011+0000","last_active":"2024-03-30T13:27:33.771146+0000","last_peered":"2024-03-30T13:27:33.771146+0000","last_clean":"2024-03-30T13:27:33.771146+0000","last_became_active":"2024-03-30T13:25:20.549465+0000","last_became_peered":"2024-03-30T13:25:20.549465+0000","last_unstale":"2024-03-30T13:27:33.771146+0000","last_undegraded":"2024-03-30T13:27:33.771146+0000","last_fullsized":"2024-03-30T13:27:33.771146+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:27:28.611732+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:27:28.611732+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:50.237192+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":842,"reported_epoch":888,"state":"active+clean","last_fresh":"2024-03-30T13:27:34.420547+0000","last_change":"2024-03-30T13:26:35.032086+0000","last_active":"2024-03-30T13:27:34.420547+0000","last_peered":"2024-03-30T13:27:34.420547+0000","last_clean":"2024-03-30T13:27:34.420547+0000","last_became_active":"2024-03-30T13:25:20.549427+0000","last_became_peered":"2024-03-30T13:25:20.549427+0000","last_unstale":"2024-03-30T13:27:34.420547+0000","last_undegraded":"2024-03-30T13:27:34.420547+0000","last_fullsized":"2024-03-30T13:27:34.420547+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:26:35.031641+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:26:35.031641+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:42.633724+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":122,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-30T13:27:35.791735+0000","last_change":"2024-03-30T13:26:25.741558+0000","last_active":"2024-03-30T13:27:35.791735+0000","last_peered":"2024-03-30T13:27:35.791735+0000","last_clean":"2024-03-30T13:27:35.791735+0000","last_became_active":"2024-03-30T13:26:16.578484+0000","last_became_peered":"2024-03-30T13:26:16.578484+0000","last_unstale":"2024-03-30T13:27:35.791735+0000","last_undegraded":"2024-03-30T13:27:35.791735+0000","last_fullsized":"2024-03-30T13:27:35.791735+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:26:25.741212+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:26:25.741212+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:47.925369+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":857,"reported_epoch":888,"state":"active+clean","last_fresh":"2024-03-30T13:27:34.420588+0000","last_change":"2024-03-30T13:26:09.903819+0000","last_active":"2024-03-30T13:27:34.420588+0000","last_peered":"2024-03-30T13:27:34.420588+0000","last_clean":"2024-03-30T13:27:34.420588+0000","last_became_active":"2024-03-30T13:25:20.556235+0000","last_became_peered":"2024-03-30T13:25:20.556235+0000","last_unstale":"2024-03-30T13:27:34.420588+0000","last_undegraded":"2024-03-30T13:27:34.420588+0000","last_fullsized":"2024-03-30T13:27:34.420588+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:26:09.903568+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_clean_scrub_stamp":"2024-03-30T13:26:09.903568+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:27:35.258779+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":1108,"reported_epoch":890,"state":"active+clean","last_fresh":"2024-03-30T13:27:36.045790+0000","last_change":"2024-03-30T13:27:30.772468+0000","last_active":"2024-03-30T13:27:36.045790+0000","last_peered":"2024-03-30T13:27:36.045790+0000","last_clean":"2024-03-30T13:27:36.045790+0000","last_became_active":"2024-03-30T13:27:07.478858+0000","last_became_peered":"2024-03-30T13:27:07.478858+0000","last_unstale":"2024-03-30T13:27:36.045790+0000","last_undegraded":"2024-03-30T13:27:36.045790+0000","last_fullsized":"2024-03-30T13:27:36.045790+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:27:30.772331+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:27:30.772331+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:38.830587+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":276723559,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387782,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59964,"kb_used_data":22760,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94311876,"statfs":{"total":96636764160,"available":96575361024,"internally_reserved":0,"allocated":23306240,"data_stored":19022299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":816,"seq":3504693313565,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69288,"kb_used_data":27480,"kb_used_omap":12,"kb_used_meta":41779,"kb_avail":94302552,"statfs":{"total":96636764160,"available":96565813248,"internally_reserved":0,"allocated":28139520,"data_stored":23869181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12654,"internal_metadata":42782354},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":794,"seq":3410204033064,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59784,"kb_used_data":19256,"kb_used_omap":15,"kb_used_meta":40496,"kb_avail":94312056,"statfs":{"total":96636764160,"available":96575545344,"internally_reserved":0,"allocated":19718144,"data_stored":15406999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15941,"internal_metadata":41468347},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509828,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53476,"kb_used_data":13976,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94318364,"statfs":{"total":96636764160,"available":96582004736,"internally_reserved":0,"allocated":14311424,"data_stored":10027483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":861,"seq":3697966841865,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50900,"kb_used_data":13252,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94320940,"statfs":{"total":96636764160,"available":96584642560,"internally_reserved":0,"allocated":13570048,"data_stored":9286107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12649,"internal_metadata":38522519},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509829,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56480,"kb_used_data":21140,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315360,"statfs":{"total":96636764160,"available":96578928640,"internally_reserved":0,"allocated":21647360,"data_stored":17370220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":837,"seq":3594887626771,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58748,"kb_used_data":17708,"kb_used_omap":17,"kb_used_meta":41006,"kb_avail":94313092,"statfs":{"total":96636764160,"available":96576606208,"internally_reserved":0,"allocated":18132992,"data_stored":13842250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17588,"internal_metadata":41990988},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570172,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56936,"kb_used_data":20176,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94314904,"statfs":{"total":96636764160,"available":96578461696,"internally_reserved":0,"allocated":20660224,"data_stored":16362681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:27:39.503 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-30T13:27:44.503 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-03-30T13:27:44.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:44.628+0000 7fde48d21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:44.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:44.640+0000 7fde48d21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:44.662 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a: {} 2024-03-30T13:27:44.662 INFO:teuthology.orchestra.run.smithi067.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-30T13:27:44.680 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b: {} 2024-03-30T13:27:44.680 INFO:teuthology.orchestra.run.smithi067.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-30T13:27:44.697 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c: {} 2024-03-30T13:27:44.697 INFO:teuthology.orchestra.run.smithi067.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-30T13:27:44.717 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-03-30T13:27:44.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:44.828+0000 7f8bb2463640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:44.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:44.844+0000 7f8bb2463640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:45.884 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.0 to 1 (10000) 2024-03-30T13:27:45.909 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-03-30T13:27:46.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:46.032+0000 7ff59f618640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:46.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:46.048+0000 7ff59f618640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:46.881 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-30T13:27:46.911 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-03-30T13:27:47.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:47.028+0000 7f15d4257640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:47.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:47.044+0000 7f15d4257640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:47.905 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.2 to 1 (10000) 2024-03-30T13:27:47.929 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-03-30T13:27:48.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:48.047+0000 7f8a361f0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:48.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:48.063+0000 7f8a361f0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:48.919 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-30T13:27:48.945 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-03-30T13:27:49.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:49.063+0000 7f9cd1b63640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:49.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:49.079+0000 7f9cd1b63640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:49.933 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.6 to 1 (10000) 2024-03-30T13:27:49.957 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-03-30T13:27:50.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:50.071+0000 7f1bb1f11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:50.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:50.087+0000 7f1bb1f11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:50.942 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-30T13:27:50.971 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-03-30T13:27:51.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:51.095+0000 7fdde1517640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:51.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:51.111+0000 7fdde1517640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:51.947 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.3 to 1 (10000) 2024-03-30T13:27:51.970 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-03-30T13:27:52.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:52.091+0000 7fbe6d4cd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:52.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:52.107+0000 7fbe6d4cd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:52.971 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-30T13:27:52.995 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-03-30T13:27:53.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:53.115+0000 7f8b5276c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:53.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:53.131+0000 7f8b5276c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:53.975 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.4 to 1 (10000) 2024-03-30T13:27:54.001 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-03-30T13:27:54.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:54.119+0000 7f73a066c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:54.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:54.135+0000 7f73a066c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:54.997 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-30T13:27:55.025 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-03-30T13:27:55.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:55.143+0000 7f6b4d38a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:55.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:55.163+0000 7f6b4d38a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:56.002 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.5 to 1 (10000) 2024-03-30T13:27:56.028 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-03-30T13:27:56.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:56.139+0000 7ff3dacf8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:56.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:56.155+0000 7ff3dacf8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:57.010 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-30T13:27:57.037 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-03-30T13:27:57.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:57.155+0000 7fc572387640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:57.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:57.171+0000 7fc572387640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:58.021 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.7 to 1 (10000) 2024-03-30T13:27:58.050 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-03-30T13:27:58.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:58.171+0000 7f7f41d83640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:58.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:58.187+0000 7f7f41d83640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:59.040 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-30T13:27:59.064 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-03-30T13:27:59.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:59.183+0000 7f21cfa9e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:27:59.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:27:59.199+0000 7f21cfa9e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:00.054 INFO:teuthology.orchestra.run.smithi067.stderr:reweighted osd.1 to 1 (10000) 2024-03-30T13:28:00.078 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-03-30T13:28:00.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:00.199+0000 7f1099a6f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:00.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:00.215+0000 7f1099a6f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:01.059 INFO:teuthology.orchestra.run.smithi067.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-30T13:28:01.086 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-03-30T13:28:01.086 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-03-30T13:28:01.087 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-03-30T13:28:01.087 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-03-30T13:28:01.087 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-30T13:28:01.087 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:28:01.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:01.203+0000 7f6cff028640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:01.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:01.215+0000 7f6cff028640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:01.393 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:28:01.393 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":911,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:28:01.059249+0000","last_up_change":"2024-03-30T13:27:06.453839+0000","last_in_change":"2024-03-30T13:27:06.453839+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":42,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":787,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":769,"last_clean_end":854,"up_from":861,"up_thru":861,"down_at":855,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2045515420},{"type":"v1","addr":"172.21.15.67:6818","nonce":2045515420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2045515420},{"type":"v1","addr":"172.21.15.67:6821","nonce":2045515420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2045515420},{"type":"v1","addr":"172.21.15.67:6828","nonce":2045515420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2045515420},{"type":"v1","addr":"172.21.15.67:6825","nonce":2045515420}]},"public_addr":"172.21.15.67:6818/2045515420","cluster_addr":"172.21.15.67:6821/2045515420","heartbeat_back_addr":"172.21.15.67:6828/2045515420","heartbeat_front_addr":"172.21.15.67:6825/2045515420","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":855,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":699,"last_clean_end":738,"up_from":747,"up_thru":838,"down_at":739,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6801","nonce":3161027348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6803","nonce":3161027348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6807","nonce":3161027348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6805","nonce":3161027348}]},"public_addr":"172.21.15.67:6801/3161027348","cluster_addr":"172.21.15.67:6803/3161027348","heartbeat_back_addr":"172.21.15.67:6807/3161027348","heartbeat_front_addr":"172.21.15.67:6805/3161027348","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":677,"last_clean_end":785,"up_from":794,"up_thru":855,"down_at":786,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":167623373},{"type":"v1","addr":"172.21.15.82:6803","nonce":167623373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":167623373},{"type":"v1","addr":"172.21.15.82:6807","nonce":167623373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":167623373},{"type":"v1","addr":"172.21.15.82:6815","nonce":167623373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":167623373},{"type":"v1","addr":"172.21.15.82:6811","nonce":167623373}]},"public_addr":"172.21.15.82:6803/167623373","cluster_addr":"172.21.15.82:6807/167623373","heartbeat_back_addr":"172.21.15.82:6815/167623373","heartbeat_front_addr":"172.21.15.82:6811/167623373","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":515,"last_clean_end":810,"up_from":816,"up_thru":833,"down_at":811,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":1480534728},{"type":"v1","addr":"172.21.15.82:6817","nonce":1480534728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":1480534728},{"type":"v1","addr":"172.21.15.82:6819","nonce":1480534728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":1480534728},{"type":"v1","addr":"172.21.15.82:6823","nonce":1480534728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":1480534728},{"type":"v1","addr":"172.21.15.82:6821","nonce":1480534728}]},"public_addr":"172.21.15.82:6817/1480534728","cluster_addr":"172.21.15.82:6819/1480534728","heartbeat_back_addr":"172.21.15.82:6823/1480534728","heartbeat_front_addr":"172.21.15.82:6821/1480534728","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":629,"last_clean_end":715,"up_from":722,"up_thru":816,"down_at":716,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6801","nonce":326680821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6806","nonce":326680821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6814","nonce":326680821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6810","nonce":326680821}]},"public_addr":"172.21.15.82:6801/326680821","cluster_addr":"172.21.15.82:6806/326680821","heartbeat_back_addr":"172.21.15.82:6814/326680821","heartbeat_front_addr":"172.21.15.82:6810/326680821","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":650,"last_clean_end":832,"up_from":837,"up_thru":861,"down_at":833,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":876600018},{"type":"v1","addr":"172.21.15.82:6825","nonce":876600018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":876600018},{"type":"v1","addr":"172.21.15.82:6827","nonce":876600018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":876600018},{"type":"v1","addr":"172.21.15.82:6831","nonce":876600018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":876600018},{"type":"v1","addr":"172.21.15.82:6829","nonce":876600018}]},"public_addr":"172.21.15.82:6825/876600018","cluster_addr":"172.21.15.82:6827/876600018","heartbeat_back_addr":"172.21.15.82:6831/876600018","heartbeat_front_addr":"172.21.15.82:6829/876600018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:26:39.634058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:27:57.710645+0000","dead_epoch":855},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:27:41.623030+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:26:47.427678+0000","dead_epoch":739},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:27:20.230932+0000","dead_epoch":786},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:27:40.166425+0000","dead_epoch":811},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:27:11.815072+0000","dead_epoch":716},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:26:48.975089+0000","dead_epoch":833}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:28:01.413 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-30T13:28:01.413 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:28:01.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:01.519+0000 7fdbad88c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:01.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:01.531+0000 7fdbad88c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:01.700 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:28:01.700 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":911,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:28:01.059249+0000","last_up_change":"2024-03-30T13:27:06.453839+0000","last_in_change":"2024-03-30T13:27:06.453839+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":42,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":787,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":769,"last_clean_end":854,"up_from":861,"up_thru":861,"down_at":855,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2045515420},{"type":"v1","addr":"172.21.15.67:6818","nonce":2045515420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2045515420},{"type":"v1","addr":"172.21.15.67:6821","nonce":2045515420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2045515420},{"type":"v1","addr":"172.21.15.67:6828","nonce":2045515420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2045515420},{"type":"v1","addr":"172.21.15.67:6825","nonce":2045515420}]},"public_addr":"172.21.15.67:6818/2045515420","cluster_addr":"172.21.15.67:6821/2045515420","heartbeat_back_addr":"172.21.15.67:6828/2045515420","heartbeat_front_addr":"172.21.15.67:6825/2045515420","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":855,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":699,"last_clean_end":738,"up_from":747,"up_thru":838,"down_at":739,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6801","nonce":3161027348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6803","nonce":3161027348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6807","nonce":3161027348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6805","nonce":3161027348}]},"public_addr":"172.21.15.67:6801/3161027348","cluster_addr":"172.21.15.67:6803/3161027348","heartbeat_back_addr":"172.21.15.67:6807/3161027348","heartbeat_front_addr":"172.21.15.67:6805/3161027348","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":677,"last_clean_end":785,"up_from":794,"up_thru":855,"down_at":786,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":167623373},{"type":"v1","addr":"172.21.15.82:6803","nonce":167623373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":167623373},{"type":"v1","addr":"172.21.15.82:6807","nonce":167623373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":167623373},{"type":"v1","addr":"172.21.15.82:6815","nonce":167623373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":167623373},{"type":"v1","addr":"172.21.15.82:6811","nonce":167623373}]},"public_addr":"172.21.15.82:6803/167623373","cluster_addr":"172.21.15.82:6807/167623373","heartbeat_back_addr":"172.21.15.82:6815/167623373","heartbeat_front_addr":"172.21.15.82:6811/167623373","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":515,"last_clean_end":810,"up_from":816,"up_thru":833,"down_at":811,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":1480534728},{"type":"v1","addr":"172.21.15.82:6817","nonce":1480534728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":1480534728},{"type":"v1","addr":"172.21.15.82:6819","nonce":1480534728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":1480534728},{"type":"v1","addr":"172.21.15.82:6823","nonce":1480534728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":1480534728},{"type":"v1","addr":"172.21.15.82:6821","nonce":1480534728}]},"public_addr":"172.21.15.82:6817/1480534728","cluster_addr":"172.21.15.82:6819/1480534728","heartbeat_back_addr":"172.21.15.82:6823/1480534728","heartbeat_front_addr":"172.21.15.82:6821/1480534728","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":629,"last_clean_end":715,"up_from":722,"up_thru":816,"down_at":716,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6801","nonce":326680821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6806","nonce":326680821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6814","nonce":326680821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6810","nonce":326680821}]},"public_addr":"172.21.15.82:6801/326680821","cluster_addr":"172.21.15.82:6806/326680821","heartbeat_back_addr":"172.21.15.82:6814/326680821","heartbeat_front_addr":"172.21.15.82:6810/326680821","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":650,"last_clean_end":832,"up_from":837,"up_thru":861,"down_at":833,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":876600018},{"type":"v1","addr":"172.21.15.82:6825","nonce":876600018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":876600018},{"type":"v1","addr":"172.21.15.82:6827","nonce":876600018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":876600018},{"type":"v1","addr":"172.21.15.82:6831","nonce":876600018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":876600018},{"type":"v1","addr":"172.21.15.82:6829","nonce":876600018}]},"public_addr":"172.21.15.82:6825/876600018","cluster_addr":"172.21.15.82:6827/876600018","heartbeat_back_addr":"172.21.15.82:6831/876600018","heartbeat_front_addr":"172.21.15.82:6829/876600018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:26:39.634058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:27:57.710645+0000","dead_epoch":855},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:27:41.623030+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:26:47.427678+0000","dead_epoch":739},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:27:20.230932+0000","dead_epoch":786},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:27:40.166425+0000","dead_epoch":811},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:27:11.815072+0000","dead_epoch":716},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:26:48.975089+0000","dead_epoch":833}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:28:01.719 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-30T13:28:01.719 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-30T13:28:01.720 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-30T13:28:01.720 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-30T13:28:01.720 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-30T13:28:01.721 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-30T13:28:01.721 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-30T13:28:01.722 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-30T13:28:01.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:01.883+0000 7f82494eb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:01.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:01.891+0000 7f5f8db55640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:01.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:01.891+0000 7f92b2744640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:01.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:01.891+0000 7f6d60405640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:01.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:01.891+0000 7fe541800640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:01.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:01.891+0000 7fd983d11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:01.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:01.895+0000 7f623603a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:01.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:01.899+0000 7f6ab146b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:01.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:01.907+0000 7f82494eb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:01.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:01.907+0000 7f5f8db55640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:01.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:01.907+0000 7f6d60405640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:01.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:01.911+0000 7fe541800640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:01.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:01.911+0000 7fd983d11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:01.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:01.911+0000 7f92b2744640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:01.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:01.919+0000 7f6234dd8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:01.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:01.919+0000 7f6ab146b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:01.980 INFO:teuthology.orchestra.run.smithi067.stdout:3504693313571 2024-03-30T13:28:01.980 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-30T13:28:01.984 INFO:teuthology.orchestra.run.smithi067.stdout:64424509835 2024-03-30T13:28:01.984 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-30T13:28:01.991 INFO:teuthology.orchestra.run.smithi067.stdout:64424509834 2024-03-30T13:28:01.991 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-30T13:28:01.992 INFO:teuthology.orchestra.run.smithi067.stdout:3208340570178 2024-03-30T13:28:01.993 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-30T13:28:02.000 INFO:teuthology.orchestra.run.smithi067.stdout:3100966387788 2024-03-30T13:28:02.001 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-30T13:28:02.002 INFO:teuthology.orchestra.run.smithi067.stdout:3697966841871 2024-03-30T13:28:02.002 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-30T13:28:02.010 INFO:teuthology.orchestra.run.smithi067.stdout:3410204033069 2024-03-30T13:28:02.010 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-30T13:28:02.012 INFO:teuthology.orchestra.run.smithi067.stdout:3594887626777 2024-03-30T13:28:02.013 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-30T13:28:02.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:02.127+0000 7f629a7c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:02.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:02.147+0000 7f629a7c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:02.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:02.143+0000 7f7e1754e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:02.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:02.155+0000 7fa5af84f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:02.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:02.167+0000 7f7e1754e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:02.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:02.175+0000 7fa5af84f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:02.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:02.171+0000 7f858892b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:02.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:02.175+0000 7fccfcd1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:02.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:02.175+0000 7f032eb17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:02.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:02.183+0000 7f5487f8e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:02.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:02.195+0000 7fccfcd1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:02.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:02.203+0000 7f5486d2c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:02.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:02.207+0000 7f032eb17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:02.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:02.207+0000 7f858359e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:02.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:02.211+0000 7fed49986640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:02.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:02.227+0000 7fed49986640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:02.490 INFO:teuthology.orchestra.run.smithi067.stdout:64424509834 2024-03-30T13:28:02.514 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509835 got 64424509834 for osd.0 2024-03-30T13:28:02.516 INFO:teuthology.orchestra.run.smithi067.stdout:3504693313570 2024-03-30T13:28:02.534 INFO:tasks.ceph.ceph_manager.ceph:need seq 3504693313571 got 3504693313570 for osd.5 2024-03-30T13:28:02.534 INFO:teuthology.orchestra.run.smithi067.stdout:3100966387786 2024-03-30T13:28:02.535 INFO:teuthology.orchestra.run.smithi067.stdout:64424509833 2024-03-30T13:28:02.557 INFO:teuthology.orchestra.run.smithi067.stdout:3697966841869 2024-03-30T13:28:02.560 INFO:tasks.ceph.ceph_manager.ceph:need seq 3100966387788 got 3100966387786 for osd.6 2024-03-30T13:28:02.562 INFO:teuthology.orchestra.run.smithi067.stdout:3410204033068 2024-03-30T13:28:02.563 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509834 got 64424509833 for osd.2 2024-03-30T13:28:02.564 INFO:teuthology.orchestra.run.smithi067.stdout:3208340570176 2024-03-30T13:28:02.571 INFO:teuthology.orchestra.run.smithi067.stdout:3594887626775 2024-03-30T13:28:02.589 INFO:tasks.ceph.ceph_manager.ceph:need seq 3697966841871 got 3697966841869 for osd.1 2024-03-30T13:28:02.591 INFO:tasks.ceph.ceph_manager.ceph:need seq 3410204033069 got 3410204033068 for osd.4 2024-03-30T13:28:02.593 INFO:tasks.ceph.ceph_manager.ceph:need seq 3208340570178 got 3208340570176 for osd.3 2024-03-30T13:28:02.597 INFO:tasks.ceph.ceph_manager.ceph:need seq 3594887626777 got 3594887626775 for osd.7 2024-03-30T13:28:03.515 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-30T13:28:03.535 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-30T13:28:03.561 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-30T13:28:03.564 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-30T13:28:03.589 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-30T13:28:03.592 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-30T13:28:03.595 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-30T13:28:03.598 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-30T13:28:03.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:03.659+0000 7f318e371640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:03.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:03.683+0000 7f318e371640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:03.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:03.715+0000 7f5e33b72640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:03.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:03.719+0000 7f2ff990c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:03.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:03.735+0000 7f5e33b72640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:03.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:03.739+0000 7f2ff990c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:03.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:03.767+0000 7fdd81f02640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:03.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:03.775+0000 7fc21cdcb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:03.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:03.775+0000 7f29f425e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:03.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:03.775+0000 7f34da3b3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:03.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:03.775+0000 7ff72bdbb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:03.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:03.791+0000 7fdd80ca0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:03.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:03.795+0000 7ff72bdbb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:03.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:03.795+0000 7fc21cdcb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:03.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:03.799+0000 7f34d9151640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:03.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:03.799+0000 7f29f2ffc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:04.002 INFO:teuthology.orchestra.run.smithi067.stdout:64424509835 2024-03-30T13:28:04.024 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509835 got 64424509835 for osd.0 2024-03-30T13:28:04.024 DEBUG:teuthology.parallel:result is None 2024-03-30T13:28:04.075 INFO:teuthology.orchestra.run.smithi067.stdout:3100966387788 2024-03-30T13:28:04.094 INFO:teuthology.orchestra.run.smithi067.stdout:3504693313571 2024-03-30T13:28:04.102 INFO:teuthology.orchestra.run.smithi067.stdout:3410204033069 2024-03-30T13:28:04.122 INFO:teuthology.orchestra.run.smithi067.stdout:3208340570178 2024-03-30T13:28:04.123 INFO:tasks.ceph.ceph_manager.ceph:need seq 3410204033069 got 3410204033069 for osd.4 2024-03-30T13:28:04.124 DEBUG:teuthology.parallel:result is None 2024-03-30T13:28:04.124 INFO:tasks.ceph.ceph_manager.ceph:need seq 3100966387788 got 3100966387788 for osd.6 2024-03-30T13:28:04.124 DEBUG:teuthology.parallel:result is None 2024-03-30T13:28:04.127 INFO:tasks.ceph.ceph_manager.ceph:need seq 3504693313571 got 3504693313571 for osd.5 2024-03-30T13:28:04.127 DEBUG:teuthology.parallel:result is None 2024-03-30T13:28:04.132 INFO:teuthology.orchestra.run.smithi067.stdout:3697966841871 2024-03-30T13:28:04.141 INFO:tasks.ceph.ceph_manager.ceph:need seq 3208340570178 got 3208340570178 for osd.3 2024-03-30T13:28:04.142 DEBUG:teuthology.parallel:result is None 2024-03-30T13:28:04.144 INFO:teuthology.orchestra.run.smithi067.stdout:3594887626777 2024-03-30T13:28:04.145 INFO:teuthology.orchestra.run.smithi067.stdout:64424509834 2024-03-30T13:28:04.157 INFO:tasks.ceph.ceph_manager.ceph:need seq 3697966841871 got 3697966841871 for osd.1 2024-03-30T13:28:04.157 DEBUG:teuthology.parallel:result is None 2024-03-30T13:28:04.165 INFO:tasks.ceph.ceph_manager.ceph:need seq 3594887626777 got 3594887626777 for osd.7 2024-03-30T13:28:04.166 DEBUG:teuthology.parallel:result is None 2024-03-30T13:28:04.167 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509834 got 64424509834 for osd.2 2024-03-30T13:28:04.167 DEBUG:teuthology.parallel:result is None 2024-03-30T13:28:04.167 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-03-30T13:28:04.167 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:28:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:04.307+0000 7f7efb911640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:04.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:04.319+0000 7f7efb911640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:04.490 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:28:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:28:04.504 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1881,"stamp":"2024-03-30T13:28:04.349721+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":276723559,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":467692,"kb_used_data":157884,"kb_used_omap":77,"kb_used_meta":309618,"kb_avail":754507028,"statfs":{"total":773094113280,"available":772615196672,"internally_reserved":0,"allocated":161673216,"data_stored":126164978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":79062,"internal_metadata":317049642},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.313204"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":2037,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.336784+0000","last_change":"2024-03-30T13:27:55.467519+0000","last_active":"2024-03-30T13:28:01.336784+0000","last_peered":"2024-03-30T13:28:01.336784+0000","last_clean":"2024-03-30T13:28:01.336784+0000","last_became_active":"2024-03-30T13:25:20.549084+0000","last_became_peered":"2024-03-30T13:25:20.549084+0000","last_unstale":"2024-03-30T13:28:01.336784+0000","last_undegraded":"2024-03-30T13:28:01.336784+0000","last_fullsized":"2024-03-30T13:28:01.336784+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:27:55.467102+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:27:55.467102+0000","last_clean_scrub_stamp":"2024-03-30T13:27:55.467102+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:10.379624+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":13764735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1925,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.066368+0000","last_change":"2024-03-30T13:27:26.798457+0000","last_active":"2024-03-30T13:28:01.066368+0000","last_peered":"2024-03-30T13:28:01.066368+0000","last_clean":"2024-03-30T13:28:01.066368+0000","last_became_active":"2024-03-30T13:26:16.583003+0000","last_became_peered":"2024-03-30T13:26:16.583003+0000","last_unstale":"2024-03-30T13:28:01.066368+0000","last_undegraded":"2024-03-30T13:28:01.066368+0000","last_fullsized":"2024-03-30T13:28:01.066368+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:26.798199+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:27:26.798199+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:54.470648+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":185,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T13:28:00.469884+0000","last_change":"2024-03-30T13:27:07.484434+0000","last_active":"2024-03-30T13:28:00.469884+0000","last_peered":"2024-03-30T13:28:00.469884+0000","last_clean":"2024-03-30T13:28:00.469884+0000","last_became_active":"2024-03-30T13:27:07.482993+0000","last_became_peered":"2024-03-30T13:27:07.482993+0000","last_unstale":"2024-03-30T13:28:00.469884+0000","last_undegraded":"2024-03-30T13:28:00.469884+0000","last_fullsized":"2024-03-30T13:28:00.469884+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:26:47.672328+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:26:47.672328+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:02.109948+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":171,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T13:28:00.469815+0000","last_change":"2024-03-30T13:27:48.638120+0000","last_active":"2024-03-30T13:28:00.469815+0000","last_peered":"2024-03-30T13:28:00.469815+0000","last_clean":"2024-03-30T13:28:00.469815+0000","last_became_active":"2024-03-30T13:26:16.577250+0000","last_became_peered":"2024-03-30T13:26:16.577250+0000","last_unstale":"2024-03-30T13:28:00.469815+0000","last_undegraded":"2024-03-30T13:28:00.469815+0000","last_fullsized":"2024-03-30T13:28:00.469815+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:27:48.637751+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:27:48.637751+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:15.915191+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":839,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.098817+0000","last_change":"2024-03-30T13:27:01.991450+0000","last_active":"2024-03-30T13:28:01.098817+0000","last_peered":"2024-03-30T13:28:01.098817+0000","last_clean":"2024-03-30T13:28:01.098817+0000","last_became_active":"2024-03-30T13:25:20.556165+0000","last_became_peered":"2024-03-30T13:25:20.556165+0000","last_unstale":"2024-03-30T13:28:01.098817+0000","last_undegraded":"2024-03-30T13:28:01.098817+0000","last_fullsized":"2024-03-30T13:28:01.098817+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:01.991062+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:27:01.991062+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:06.741931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1976,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.067440+0000","last_change":"2024-03-30T13:27:39.308222+0000","last_active":"2024-03-30T13:28:01.067440+0000","last_peered":"2024-03-30T13:28:01.067440+0000","last_clean":"2024-03-30T13:28:01.067440+0000","last_became_active":"2024-03-30T13:25:20.551941+0000","last_became_peered":"2024-03-30T13:25:20.551941+0000","last_unstale":"2024-03-30T13:28:01.067440+0000","last_undegraded":"2024-03-30T13:28:01.067440+0000","last_fullsized":"2024-03-30T13:28:01.067440+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:27:39.308113+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:27:39.308113+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:47.332129+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11394981,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":178,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T13:28:00.469821+0000","last_change":"2024-03-30T13:27:12.597085+0000","last_active":"2024-03-30T13:28:00.469821+0000","last_peered":"2024-03-30T13:28:00.469821+0000","last_clean":"2024-03-30T13:28:00.469821+0000","last_became_active":"2024-03-30T13:26:16.583861+0000","last_became_peered":"2024-03-30T13:26:16.583861+0000","last_unstale":"2024-03-30T13:28:00.469821+0000","last_undegraded":"2024-03-30T13:28:00.469821+0000","last_fullsized":"2024-03-30T13:28:00.469821+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:27:12.596825+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:27:12.596825+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:25.628559+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1284,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.126491+0000","last_change":"2024-03-30T13:27:22.661584+0000","last_active":"2024-03-30T13:28:01.126491+0000","last_peered":"2024-03-30T13:28:01.126491+0000","last_clean":"2024-03-30T13:28:01.126491+0000","last_became_active":"2024-03-30T13:27:07.598881+0000","last_became_peered":"2024-03-30T13:27:07.598881+0000","last_unstale":"2024-03-30T13:28:01.126491+0000","last_undegraded":"2024-03-30T13:28:01.126491+0000","last_fullsized":"2024-03-30T13:28:01.126491+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:27:22.661428+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:27:22.661428+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:51.172824+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":2097,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.067339+0000","last_change":"2024-03-30T13:27:41.019396+0000","last_active":"2024-03-30T13:28:01.067339+0000","last_peered":"2024-03-30T13:28:01.067339+0000","last_clean":"2024-03-30T13:28:01.067339+0000","last_became_active":"2024-03-30T13:26:16.582448+0000","last_became_peered":"2024-03-30T13:26:16.582448+0000","last_unstale":"2024-03-30T13:28:01.067339+0000","last_undegraded":"2024-03-30T13:28:01.067339+0000","last_fullsized":"2024-03-30T13:28:01.067339+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:27:41.018992+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:27:41.018992+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:06.568280+0000","scrub_duration":715,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":76134720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1735,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.066182+0000","last_change":"2024-03-30T13:27:58.853482+0000","last_active":"2024-03-30T13:28:01.066182+0000","last_peered":"2024-03-30T13:28:01.066182+0000","last_clean":"2024-03-30T13:28:01.066182+0000","last_became_active":"2024-03-30T13:24:29.442802+0000","last_became_peered":"2024-03-30T13:24:29.442802+0000","last_unstale":"2024-03-30T13:28:01.066182+0000","last_undegraded":"2024-03-30T13:28:01.066182+0000","last_fullsized":"2024-03-30T13:28:01.066182+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:58.853242+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:27:58.853242+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:12.971711+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":2044,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.336514+0000","last_change":"2024-03-30T13:27:17.528668+0000","last_active":"2024-03-30T13:28:01.336514+0000","last_peered":"2024-03-30T13:28:01.336514+0000","last_clean":"2024-03-30T13:28:01.336514+0000","last_became_active":"2024-03-30T13:24:29.447202+0000","last_became_peered":"2024-03-30T13:24:29.447202+0000","last_unstale":"2024-03-30T13:28:01.336514+0000","last_undegraded":"2024-03-30T13:28:01.336514+0000","last_fullsized":"2024-03-30T13:28:01.336514+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:27:17.528414+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:27:17.528414+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:25.562324+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":2138,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.336546+0000","last_change":"2024-03-30T13:27:28.612011+0000","last_active":"2024-03-30T13:28:01.336546+0000","last_peered":"2024-03-30T13:28:01.336546+0000","last_clean":"2024-03-30T13:28:01.336546+0000","last_became_active":"2024-03-30T13:25:20.549465+0000","last_became_peered":"2024-03-30T13:25:20.549465+0000","last_unstale":"2024-03-30T13:28:01.336546+0000","last_undegraded":"2024-03-30T13:28:01.336546+0000","last_fullsized":"2024-03-30T13:28:01.336546+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:27:28.611732+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:27:28.611732+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:50.237192+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":892,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.098817+0000","last_change":"2024-03-30T13:27:42.924031+0000","last_active":"2024-03-30T13:28:01.098817+0000","last_peered":"2024-03-30T13:28:01.098817+0000","last_clean":"2024-03-30T13:28:01.098817+0000","last_became_active":"2024-03-30T13:25:20.549427+0000","last_became_peered":"2024-03-30T13:25:20.549427+0000","last_unstale":"2024-03-30T13:28:01.098817+0000","last_undegraded":"2024-03-30T13:28:01.098817+0000","last_fullsized":"2024-03-30T13:28:01.098817+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:27:42.923778+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:27:42.923778+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:12.595140+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":169,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T13:28:00.469819+0000","last_change":"2024-03-30T13:27:51.677273+0000","last_active":"2024-03-30T13:28:00.469819+0000","last_peered":"2024-03-30T13:28:00.469819+0000","last_clean":"2024-03-30T13:28:00.469819+0000","last_became_active":"2024-03-30T13:26:16.578484+0000","last_became_peered":"2024-03-30T13:26:16.578484+0000","last_unstale":"2024-03-30T13:28:00.469819+0000","last_undegraded":"2024-03-30T13:28:00.469819+0000","last_fullsized":"2024-03-30T13:28:00.469819+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:51.677029+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:27:51.677029+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:14.237232+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":907,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.098821+0000","last_change":"2024-03-30T13:27:35.881662+0000","last_active":"2024-03-30T13:28:01.098821+0000","last_peered":"2024-03-30T13:28:01.098821+0000","last_clean":"2024-03-30T13:28:01.098821+0000","last_became_active":"2024-03-30T13:25:20.556235+0000","last_became_peered":"2024-03-30T13:25:20.556235+0000","last_unstale":"2024-03-30T13:28:01.098821+0000","last_undegraded":"2024-03-30T13:28:01.098821+0000","last_fullsized":"2024-03-30T13:28:01.098821+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:27:35.881272+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_clean_scrub_stamp":"2024-03-30T13:27:35.881272+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:48.834909+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":1147,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.126491+0000","last_change":"2024-03-30T13:27:30.772468+0000","last_active":"2024-03-30T13:28:01.126491+0000","last_peered":"2024-03-30T13:28:01.126491+0000","last_clean":"2024-03-30T13:28:01.126491+0000","last_became_active":"2024-03-30T13:27:07.478858+0000","last_became_peered":"2024-03-30T13:27:07.478858+0000","last_unstale":"2024-03-30T13:28:01.126491+0000","last_undegraded":"2024-03-30T13:28:01.126491+0000","last_fullsized":"2024-03-30T13:28:01.126491+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:27:30.772331+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:27:30.772331+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:38.830587+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":276723559,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387788,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60232,"kb_used_data":23036,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94311608,"statfs":{"total":96636764160,"available":96575086592,"internally_reserved":0,"allocated":23588864,"data_stored":19148634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":816,"seq":3504693313571,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69588,"kb_used_data":27780,"kb_used_omap":12,"kb_used_meta":41779,"kb_avail":94302252,"statfs":{"total":96636764160,"available":96565506048,"internally_reserved":0,"allocated":28446720,"data_stored":24006490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12654,"internal_metadata":42782354},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":794,"seq":3410204033070,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60012,"kb_used_data":19484,"kb_used_omap":15,"kb_used_meta":40496,"kb_avail":94311828,"statfs":{"total":96636764160,"available":96575311872,"internally_reserved":0,"allocated":19951616,"data_stored":15511386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15941,"internal_metadata":41468347},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509834,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53752,"kb_used_data":14252,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94318088,"statfs":{"total":96636764160,"available":96581722112,"internally_reserved":0,"allocated":14594048,"data_stored":10153818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":861,"seq":3697966841871,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51180,"kb_used_data":13528,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94320660,"statfs":{"total":96636764160,"available":96584355840,"internally_reserved":0,"allocated":13852672,"data_stored":9412442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12649,"internal_metadata":38522519},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509835,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56756,"kb_used_data":21416,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315084,"statfs":{"total":96636764160,"available":96578646016,"internally_reserved":0,"allocated":21929984,"data_stored":17496555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":837,"seq":3594887626777,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58996,"kb_used_data":17960,"kb_used_omap":17,"kb_used_meta":41006,"kb_avail":94312844,"statfs":{"total":96636764160,"available":96576352256,"internally_reserved":0,"allocated":18391040,"data_stored":13957611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17588,"internal_metadata":41990988},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570178,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57176,"kb_used_data":20428,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94314664,"statfs":{"total":96636764160,"available":96578215936,"internally_reserved":0,"allocated":20918272,"data_stored":16478042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:28:04.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:28:04.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:04.607+0000 7fd767d70640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:04.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:04.623+0000 7fd767d70640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:04.789 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:28:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:28:04.806 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1881,"stamp":"2024-03-30T13:28:04.349721+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":276723559,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":467692,"kb_used_data":157884,"kb_used_omap":77,"kb_used_meta":309618,"kb_avail":754507028,"statfs":{"total":773094113280,"available":772615196672,"internally_reserved":0,"allocated":161673216,"data_stored":126164978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":79062,"internal_metadata":317049642},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.313204"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":2037,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.336784+0000","last_change":"2024-03-30T13:27:55.467519+0000","last_active":"2024-03-30T13:28:01.336784+0000","last_peered":"2024-03-30T13:28:01.336784+0000","last_clean":"2024-03-30T13:28:01.336784+0000","last_became_active":"2024-03-30T13:25:20.549084+0000","last_became_peered":"2024-03-30T13:25:20.549084+0000","last_unstale":"2024-03-30T13:28:01.336784+0000","last_undegraded":"2024-03-30T13:28:01.336784+0000","last_fullsized":"2024-03-30T13:28:01.336784+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:27:55.467102+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:27:55.467102+0000","last_clean_scrub_stamp":"2024-03-30T13:27:55.467102+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:10.379624+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":13764735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1925,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.066368+0000","last_change":"2024-03-30T13:27:26.798457+0000","last_active":"2024-03-30T13:28:01.066368+0000","last_peered":"2024-03-30T13:28:01.066368+0000","last_clean":"2024-03-30T13:28:01.066368+0000","last_became_active":"2024-03-30T13:26:16.583003+0000","last_became_peered":"2024-03-30T13:26:16.583003+0000","last_unstale":"2024-03-30T13:28:01.066368+0000","last_undegraded":"2024-03-30T13:28:01.066368+0000","last_fullsized":"2024-03-30T13:28:01.066368+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:26.798199+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:27:26.798199+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:54.470648+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":185,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T13:28:00.469884+0000","last_change":"2024-03-30T13:27:07.484434+0000","last_active":"2024-03-30T13:28:00.469884+0000","last_peered":"2024-03-30T13:28:00.469884+0000","last_clean":"2024-03-30T13:28:00.469884+0000","last_became_active":"2024-03-30T13:27:07.482993+0000","last_became_peered":"2024-03-30T13:27:07.482993+0000","last_unstale":"2024-03-30T13:28:00.469884+0000","last_undegraded":"2024-03-30T13:28:00.469884+0000","last_fullsized":"2024-03-30T13:28:00.469884+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:26:47.672328+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:26:47.672328+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:02.109948+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":171,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T13:28:00.469815+0000","last_change":"2024-03-30T13:27:48.638120+0000","last_active":"2024-03-30T13:28:00.469815+0000","last_peered":"2024-03-30T13:28:00.469815+0000","last_clean":"2024-03-30T13:28:00.469815+0000","last_became_active":"2024-03-30T13:26:16.577250+0000","last_became_peered":"2024-03-30T13:26:16.577250+0000","last_unstale":"2024-03-30T13:28:00.469815+0000","last_undegraded":"2024-03-30T13:28:00.469815+0000","last_fullsized":"2024-03-30T13:28:00.469815+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:27:48.637751+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:27:48.637751+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:15.915191+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":839,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.098817+0000","last_change":"2024-03-30T13:27:01.991450+0000","last_active":"2024-03-30T13:28:01.098817+0000","last_peered":"2024-03-30T13:28:01.098817+0000","last_clean":"2024-03-30T13:28:01.098817+0000","last_became_active":"2024-03-30T13:25:20.556165+0000","last_became_peered":"2024-03-30T13:25:20.556165+0000","last_unstale":"2024-03-30T13:28:01.098817+0000","last_undegraded":"2024-03-30T13:28:01.098817+0000","last_fullsized":"2024-03-30T13:28:01.098817+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:01.991062+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:27:01.991062+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:06.741931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1976,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.067440+0000","last_change":"2024-03-30T13:27:39.308222+0000","last_active":"2024-03-30T13:28:01.067440+0000","last_peered":"2024-03-30T13:28:01.067440+0000","last_clean":"2024-03-30T13:28:01.067440+0000","last_became_active":"2024-03-30T13:25:20.551941+0000","last_became_peered":"2024-03-30T13:25:20.551941+0000","last_unstale":"2024-03-30T13:28:01.067440+0000","last_undegraded":"2024-03-30T13:28:01.067440+0000","last_fullsized":"2024-03-30T13:28:01.067440+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:27:39.308113+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:27:39.308113+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:47.332129+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11394981,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":178,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T13:28:00.469821+0000","last_change":"2024-03-30T13:27:12.597085+0000","last_active":"2024-03-30T13:28:00.469821+0000","last_peered":"2024-03-30T13:28:00.469821+0000","last_clean":"2024-03-30T13:28:00.469821+0000","last_became_active":"2024-03-30T13:26:16.583861+0000","last_became_peered":"2024-03-30T13:26:16.583861+0000","last_unstale":"2024-03-30T13:28:00.469821+0000","last_undegraded":"2024-03-30T13:28:00.469821+0000","last_fullsized":"2024-03-30T13:28:00.469821+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:27:12.596825+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:27:12.596825+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:25.628559+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1284,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.126491+0000","last_change":"2024-03-30T13:27:22.661584+0000","last_active":"2024-03-30T13:28:01.126491+0000","last_peered":"2024-03-30T13:28:01.126491+0000","last_clean":"2024-03-30T13:28:01.126491+0000","last_became_active":"2024-03-30T13:27:07.598881+0000","last_became_peered":"2024-03-30T13:27:07.598881+0000","last_unstale":"2024-03-30T13:28:01.126491+0000","last_undegraded":"2024-03-30T13:28:01.126491+0000","last_fullsized":"2024-03-30T13:28:01.126491+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:27:22.661428+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:27:22.661428+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:51.172824+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":2097,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.067339+0000","last_change":"2024-03-30T13:27:41.019396+0000","last_active":"2024-03-30T13:28:01.067339+0000","last_peered":"2024-03-30T13:28:01.067339+0000","last_clean":"2024-03-30T13:28:01.067339+0000","last_became_active":"2024-03-30T13:26:16.582448+0000","last_became_peered":"2024-03-30T13:26:16.582448+0000","last_unstale":"2024-03-30T13:28:01.067339+0000","last_undegraded":"2024-03-30T13:28:01.067339+0000","last_fullsized":"2024-03-30T13:28:01.067339+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:27:41.018992+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:27:41.018992+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:06.568280+0000","scrub_duration":715,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":76134720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1735,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.066182+0000","last_change":"2024-03-30T13:27:58.853482+0000","last_active":"2024-03-30T13:28:01.066182+0000","last_peered":"2024-03-30T13:28:01.066182+0000","last_clean":"2024-03-30T13:28:01.066182+0000","last_became_active":"2024-03-30T13:24:29.442802+0000","last_became_peered":"2024-03-30T13:24:29.442802+0000","last_unstale":"2024-03-30T13:28:01.066182+0000","last_undegraded":"2024-03-30T13:28:01.066182+0000","last_fullsized":"2024-03-30T13:28:01.066182+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:58.853242+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:27:58.853242+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:12.971711+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":2044,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.336514+0000","last_change":"2024-03-30T13:27:17.528668+0000","last_active":"2024-03-30T13:28:01.336514+0000","last_peered":"2024-03-30T13:28:01.336514+0000","last_clean":"2024-03-30T13:28:01.336514+0000","last_became_active":"2024-03-30T13:24:29.447202+0000","last_became_peered":"2024-03-30T13:24:29.447202+0000","last_unstale":"2024-03-30T13:28:01.336514+0000","last_undegraded":"2024-03-30T13:28:01.336514+0000","last_fullsized":"2024-03-30T13:28:01.336514+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:27:17.528414+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:27:17.528414+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:25.562324+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":2138,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.336546+0000","last_change":"2024-03-30T13:27:28.612011+0000","last_active":"2024-03-30T13:28:01.336546+0000","last_peered":"2024-03-30T13:28:01.336546+0000","last_clean":"2024-03-30T13:28:01.336546+0000","last_became_active":"2024-03-30T13:25:20.549465+0000","last_became_peered":"2024-03-30T13:25:20.549465+0000","last_unstale":"2024-03-30T13:28:01.336546+0000","last_undegraded":"2024-03-30T13:28:01.336546+0000","last_fullsized":"2024-03-30T13:28:01.336546+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:27:28.611732+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:27:28.611732+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:50.237192+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":892,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.098817+0000","last_change":"2024-03-30T13:27:42.924031+0000","last_active":"2024-03-30T13:28:01.098817+0000","last_peered":"2024-03-30T13:28:01.098817+0000","last_clean":"2024-03-30T13:28:01.098817+0000","last_became_active":"2024-03-30T13:25:20.549427+0000","last_became_peered":"2024-03-30T13:25:20.549427+0000","last_unstale":"2024-03-30T13:28:01.098817+0000","last_undegraded":"2024-03-30T13:28:01.098817+0000","last_fullsized":"2024-03-30T13:28:01.098817+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:27:42.923778+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:27:42.923778+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:12.595140+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":169,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T13:28:00.469819+0000","last_change":"2024-03-30T13:27:51.677273+0000","last_active":"2024-03-30T13:28:00.469819+0000","last_peered":"2024-03-30T13:28:00.469819+0000","last_clean":"2024-03-30T13:28:00.469819+0000","last_became_active":"2024-03-30T13:26:16.578484+0000","last_became_peered":"2024-03-30T13:26:16.578484+0000","last_unstale":"2024-03-30T13:28:00.469819+0000","last_undegraded":"2024-03-30T13:28:00.469819+0000","last_fullsized":"2024-03-30T13:28:00.469819+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:51.677029+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:27:51.677029+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:14.237232+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":907,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.098821+0000","last_change":"2024-03-30T13:27:35.881662+0000","last_active":"2024-03-30T13:28:01.098821+0000","last_peered":"2024-03-30T13:28:01.098821+0000","last_clean":"2024-03-30T13:28:01.098821+0000","last_became_active":"2024-03-30T13:25:20.556235+0000","last_became_peered":"2024-03-30T13:25:20.556235+0000","last_unstale":"2024-03-30T13:28:01.098821+0000","last_undegraded":"2024-03-30T13:28:01.098821+0000","last_fullsized":"2024-03-30T13:28:01.098821+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:27:35.881272+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_clean_scrub_stamp":"2024-03-30T13:27:35.881272+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:48.834909+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":1147,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.126491+0000","last_change":"2024-03-30T13:27:30.772468+0000","last_active":"2024-03-30T13:28:01.126491+0000","last_peered":"2024-03-30T13:28:01.126491+0000","last_clean":"2024-03-30T13:28:01.126491+0000","last_became_active":"2024-03-30T13:27:07.478858+0000","last_became_peered":"2024-03-30T13:27:07.478858+0000","last_unstale":"2024-03-30T13:28:01.126491+0000","last_undegraded":"2024-03-30T13:28:01.126491+0000","last_fullsized":"2024-03-30T13:28:01.126491+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:27:30.772331+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:27:30.772331+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:38.830587+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":276723559,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387788,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60232,"kb_used_data":23036,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94311608,"statfs":{"total":96636764160,"available":96575086592,"internally_reserved":0,"allocated":23588864,"data_stored":19148634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":816,"seq":3504693313571,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69588,"kb_used_data":27780,"kb_used_omap":12,"kb_used_meta":41779,"kb_avail":94302252,"statfs":{"total":96636764160,"available":96565506048,"internally_reserved":0,"allocated":28446720,"data_stored":24006490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12654,"internal_metadata":42782354},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":794,"seq":3410204033070,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60012,"kb_used_data":19484,"kb_used_omap":15,"kb_used_meta":40496,"kb_avail":94311828,"statfs":{"total":96636764160,"available":96575311872,"internally_reserved":0,"allocated":19951616,"data_stored":15511386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15941,"internal_metadata":41468347},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509834,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53752,"kb_used_data":14252,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94318088,"statfs":{"total":96636764160,"available":96581722112,"internally_reserved":0,"allocated":14594048,"data_stored":10153818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":861,"seq":3697966841871,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51180,"kb_used_data":13528,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94320660,"statfs":{"total":96636764160,"available":96584355840,"internally_reserved":0,"allocated":13852672,"data_stored":9412442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12649,"internal_metadata":38522519},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509835,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56756,"kb_used_data":21416,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315084,"statfs":{"total":96636764160,"available":96578646016,"internally_reserved":0,"allocated":21929984,"data_stored":17496555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":837,"seq":3594887626777,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58996,"kb_used_data":17960,"kb_used_omap":17,"kb_used_meta":41006,"kb_avail":94312844,"statfs":{"total":96636764160,"available":96576352256,"internally_reserved":0,"allocated":18391040,"data_stored":13957611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17588,"internal_metadata":41990988},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570178,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57176,"kb_used_data":20428,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94314664,"statfs":{"total":96636764160,"available":96578215936,"internally_reserved":0,"allocated":20918272,"data_stored":16478042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:28:04.807 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-03-30T13:28:04.807 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-03-30T13:28:04.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:04.911+0000 7f013c0e3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:04.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:04.927+0000 7f013c0e3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:05.132 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-30T13:28:05.142 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-30T13:28:05.142 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:28:05.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:05.243+0000 7fc9bb9d3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:05.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:05.259+0000 7fc9bb9d3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:05.430 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:28:05.430 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:28:05.448 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1881,"stamp":"2024-03-30T13:28:04.349721+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":276723559,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":467692,"kb_used_data":157884,"kb_used_omap":77,"kb_used_meta":309618,"kb_avail":754507028,"statfs":{"total":773094113280,"available":772615196672,"internally_reserved":0,"allocated":161673216,"data_stored":126164978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":79062,"internal_metadata":317049642},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.313204"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":2037,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.336784+0000","last_change":"2024-03-30T13:27:55.467519+0000","last_active":"2024-03-30T13:28:01.336784+0000","last_peered":"2024-03-30T13:28:01.336784+0000","last_clean":"2024-03-30T13:28:01.336784+0000","last_became_active":"2024-03-30T13:25:20.549084+0000","last_became_peered":"2024-03-30T13:25:20.549084+0000","last_unstale":"2024-03-30T13:28:01.336784+0000","last_undegraded":"2024-03-30T13:28:01.336784+0000","last_fullsized":"2024-03-30T13:28:01.336784+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:27:55.467102+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:27:55.467102+0000","last_clean_scrub_stamp":"2024-03-30T13:27:55.467102+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:10.379624+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":13764735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1925,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.066368+0000","last_change":"2024-03-30T13:27:26.798457+0000","last_active":"2024-03-30T13:28:01.066368+0000","last_peered":"2024-03-30T13:28:01.066368+0000","last_clean":"2024-03-30T13:28:01.066368+0000","last_became_active":"2024-03-30T13:26:16.583003+0000","last_became_peered":"2024-03-30T13:26:16.583003+0000","last_unstale":"2024-03-30T13:28:01.066368+0000","last_undegraded":"2024-03-30T13:28:01.066368+0000","last_fullsized":"2024-03-30T13:28:01.066368+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:26.798199+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:27:26.798199+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:54.470648+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":185,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T13:28:00.469884+0000","last_change":"2024-03-30T13:27:07.484434+0000","last_active":"2024-03-30T13:28:00.469884+0000","last_peered":"2024-03-30T13:28:00.469884+0000","last_clean":"2024-03-30T13:28:00.469884+0000","last_became_active":"2024-03-30T13:27:07.482993+0000","last_became_peered":"2024-03-30T13:27:07.482993+0000","last_unstale":"2024-03-30T13:28:00.469884+0000","last_undegraded":"2024-03-30T13:28:00.469884+0000","last_fullsized":"2024-03-30T13:28:00.469884+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:26:47.672328+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:26:47.672328+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:02.109948+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":171,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T13:28:00.469815+0000","last_change":"2024-03-30T13:27:48.638120+0000","last_active":"2024-03-30T13:28:00.469815+0000","last_peered":"2024-03-30T13:28:00.469815+0000","last_clean":"2024-03-30T13:28:00.469815+0000","last_became_active":"2024-03-30T13:26:16.577250+0000","last_became_peered":"2024-03-30T13:26:16.577250+0000","last_unstale":"2024-03-30T13:28:00.469815+0000","last_undegraded":"2024-03-30T13:28:00.469815+0000","last_fullsized":"2024-03-30T13:28:00.469815+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:27:48.637751+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:27:48.637751+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:15.915191+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":839,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.098817+0000","last_change":"2024-03-30T13:27:01.991450+0000","last_active":"2024-03-30T13:28:01.098817+0000","last_peered":"2024-03-30T13:28:01.098817+0000","last_clean":"2024-03-30T13:28:01.098817+0000","last_became_active":"2024-03-30T13:25:20.556165+0000","last_became_peered":"2024-03-30T13:25:20.556165+0000","last_unstale":"2024-03-30T13:28:01.098817+0000","last_undegraded":"2024-03-30T13:28:01.098817+0000","last_fullsized":"2024-03-30T13:28:01.098817+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:01.991062+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:27:01.991062+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:06.741931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1976,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.067440+0000","last_change":"2024-03-30T13:27:39.308222+0000","last_active":"2024-03-30T13:28:01.067440+0000","last_peered":"2024-03-30T13:28:01.067440+0000","last_clean":"2024-03-30T13:28:01.067440+0000","last_became_active":"2024-03-30T13:25:20.551941+0000","last_became_peered":"2024-03-30T13:25:20.551941+0000","last_unstale":"2024-03-30T13:28:01.067440+0000","last_undegraded":"2024-03-30T13:28:01.067440+0000","last_fullsized":"2024-03-30T13:28:01.067440+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:27:39.308113+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:27:39.308113+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:47.332129+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11394981,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":178,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T13:28:00.469821+0000","last_change":"2024-03-30T13:27:12.597085+0000","last_active":"2024-03-30T13:28:00.469821+0000","last_peered":"2024-03-30T13:28:00.469821+0000","last_clean":"2024-03-30T13:28:00.469821+0000","last_became_active":"2024-03-30T13:26:16.583861+0000","last_became_peered":"2024-03-30T13:26:16.583861+0000","last_unstale":"2024-03-30T13:28:00.469821+0000","last_undegraded":"2024-03-30T13:28:00.469821+0000","last_fullsized":"2024-03-30T13:28:00.469821+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:27:12.596825+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:27:12.596825+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:25.628559+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1284,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.126491+0000","last_change":"2024-03-30T13:27:22.661584+0000","last_active":"2024-03-30T13:28:01.126491+0000","last_peered":"2024-03-30T13:28:01.126491+0000","last_clean":"2024-03-30T13:28:01.126491+0000","last_became_active":"2024-03-30T13:27:07.598881+0000","last_became_peered":"2024-03-30T13:27:07.598881+0000","last_unstale":"2024-03-30T13:28:01.126491+0000","last_undegraded":"2024-03-30T13:28:01.126491+0000","last_fullsized":"2024-03-30T13:28:01.126491+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:27:22.661428+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:27:22.661428+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:51.172824+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":2097,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.067339+0000","last_change":"2024-03-30T13:27:41.019396+0000","last_active":"2024-03-30T13:28:01.067339+0000","last_peered":"2024-03-30T13:28:01.067339+0000","last_clean":"2024-03-30T13:28:01.067339+0000","last_became_active":"2024-03-30T13:26:16.582448+0000","last_became_peered":"2024-03-30T13:26:16.582448+0000","last_unstale":"2024-03-30T13:28:01.067339+0000","last_undegraded":"2024-03-30T13:28:01.067339+0000","last_fullsized":"2024-03-30T13:28:01.067339+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:27:41.018992+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:27:41.018992+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:06.568280+0000","scrub_duration":715,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":76134720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1735,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.066182+0000","last_change":"2024-03-30T13:27:58.853482+0000","last_active":"2024-03-30T13:28:01.066182+0000","last_peered":"2024-03-30T13:28:01.066182+0000","last_clean":"2024-03-30T13:28:01.066182+0000","last_became_active":"2024-03-30T13:24:29.442802+0000","last_became_peered":"2024-03-30T13:24:29.442802+0000","last_unstale":"2024-03-30T13:28:01.066182+0000","last_undegraded":"2024-03-30T13:28:01.066182+0000","last_fullsized":"2024-03-30T13:28:01.066182+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:58.853242+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:27:58.853242+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:12.971711+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":2044,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.336514+0000","last_change":"2024-03-30T13:27:17.528668+0000","last_active":"2024-03-30T13:28:01.336514+0000","last_peered":"2024-03-30T13:28:01.336514+0000","last_clean":"2024-03-30T13:28:01.336514+0000","last_became_active":"2024-03-30T13:24:29.447202+0000","last_became_peered":"2024-03-30T13:24:29.447202+0000","last_unstale":"2024-03-30T13:28:01.336514+0000","last_undegraded":"2024-03-30T13:28:01.336514+0000","last_fullsized":"2024-03-30T13:28:01.336514+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:27:17.528414+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:27:17.528414+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:25.562324+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":2138,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.336546+0000","last_change":"2024-03-30T13:27:28.612011+0000","last_active":"2024-03-30T13:28:01.336546+0000","last_peered":"2024-03-30T13:28:01.336546+0000","last_clean":"2024-03-30T13:28:01.336546+0000","last_became_active":"2024-03-30T13:25:20.549465+0000","last_became_peered":"2024-03-30T13:25:20.549465+0000","last_unstale":"2024-03-30T13:28:01.336546+0000","last_undegraded":"2024-03-30T13:28:01.336546+0000","last_fullsized":"2024-03-30T13:28:01.336546+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:27:28.611732+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:27:28.611732+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:50.237192+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":892,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.098817+0000","last_change":"2024-03-30T13:27:42.924031+0000","last_active":"2024-03-30T13:28:01.098817+0000","last_peered":"2024-03-30T13:28:01.098817+0000","last_clean":"2024-03-30T13:28:01.098817+0000","last_became_active":"2024-03-30T13:25:20.549427+0000","last_became_peered":"2024-03-30T13:25:20.549427+0000","last_unstale":"2024-03-30T13:28:01.098817+0000","last_undegraded":"2024-03-30T13:28:01.098817+0000","last_fullsized":"2024-03-30T13:28:01.098817+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:27:42.923778+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:27:42.923778+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:12.595140+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":169,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T13:28:00.469819+0000","last_change":"2024-03-30T13:27:51.677273+0000","last_active":"2024-03-30T13:28:00.469819+0000","last_peered":"2024-03-30T13:28:00.469819+0000","last_clean":"2024-03-30T13:28:00.469819+0000","last_became_active":"2024-03-30T13:26:16.578484+0000","last_became_peered":"2024-03-30T13:26:16.578484+0000","last_unstale":"2024-03-30T13:28:00.469819+0000","last_undegraded":"2024-03-30T13:28:00.469819+0000","last_fullsized":"2024-03-30T13:28:00.469819+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:51.677029+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:27:51.677029+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:14.237232+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":907,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.098821+0000","last_change":"2024-03-30T13:27:35.881662+0000","last_active":"2024-03-30T13:28:01.098821+0000","last_peered":"2024-03-30T13:28:01.098821+0000","last_clean":"2024-03-30T13:28:01.098821+0000","last_became_active":"2024-03-30T13:25:20.556235+0000","last_became_peered":"2024-03-30T13:25:20.556235+0000","last_unstale":"2024-03-30T13:28:01.098821+0000","last_undegraded":"2024-03-30T13:28:01.098821+0000","last_fullsized":"2024-03-30T13:28:01.098821+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:27:35.881272+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_clean_scrub_stamp":"2024-03-30T13:27:35.881272+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:48.834909+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":1147,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.126491+0000","last_change":"2024-03-30T13:27:30.772468+0000","last_active":"2024-03-30T13:28:01.126491+0000","last_peered":"2024-03-30T13:28:01.126491+0000","last_clean":"2024-03-30T13:28:01.126491+0000","last_became_active":"2024-03-30T13:27:07.478858+0000","last_became_peered":"2024-03-30T13:27:07.478858+0000","last_unstale":"2024-03-30T13:28:01.126491+0000","last_undegraded":"2024-03-30T13:28:01.126491+0000","last_fullsized":"2024-03-30T13:28:01.126491+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:27:30.772331+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:27:30.772331+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:38.830587+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":276723559,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387788,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60232,"kb_used_data":23036,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94311608,"statfs":{"total":96636764160,"available":96575086592,"internally_reserved":0,"allocated":23588864,"data_stored":19148634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":816,"seq":3504693313571,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69588,"kb_used_data":27780,"kb_used_omap":12,"kb_used_meta":41779,"kb_avail":94302252,"statfs":{"total":96636764160,"available":96565506048,"internally_reserved":0,"allocated":28446720,"data_stored":24006490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12654,"internal_metadata":42782354},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":794,"seq":3410204033070,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60012,"kb_used_data":19484,"kb_used_omap":15,"kb_used_meta":40496,"kb_avail":94311828,"statfs":{"total":96636764160,"available":96575311872,"internally_reserved":0,"allocated":19951616,"data_stored":15511386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15941,"internal_metadata":41468347},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509834,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53752,"kb_used_data":14252,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94318088,"statfs":{"total":96636764160,"available":96581722112,"internally_reserved":0,"allocated":14594048,"data_stored":10153818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":861,"seq":3697966841871,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51180,"kb_used_data":13528,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94320660,"statfs":{"total":96636764160,"available":96584355840,"internally_reserved":0,"allocated":13852672,"data_stored":9412442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12649,"internal_metadata":38522519},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509835,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56756,"kb_used_data":21416,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315084,"statfs":{"total":96636764160,"available":96578646016,"internally_reserved":0,"allocated":21929984,"data_stored":17496555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":837,"seq":3594887626777,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58996,"kb_used_data":17960,"kb_used_omap":17,"kb_used_meta":41006,"kb_avail":94312844,"statfs":{"total":96636764160,"available":96576352256,"internally_reserved":0,"allocated":18391040,"data_stored":13957611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17588,"internal_metadata":41990988},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570178,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57176,"kb_used_data":20428,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94314664,"statfs":{"total":96636764160,"available":96578215936,"internally_reserved":0,"allocated":20918272,"data_stored":16478042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:28:05.449 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:28:05.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:05.551+0000 7fb1b73aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:05.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:05.563+0000 7fb1b73aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:05.732 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:28:05.733 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:28:05.749 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1881,"stamp":"2024-03-30T13:28:04.349721+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":276723559,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":467692,"kb_used_data":157884,"kb_used_omap":77,"kb_used_meta":309618,"kb_avail":754507028,"statfs":{"total":773094113280,"available":772615196672,"internally_reserved":0,"allocated":161673216,"data_stored":126164978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":79062,"internal_metadata":317049642},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.313204"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":2037,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.336784+0000","last_change":"2024-03-30T13:27:55.467519+0000","last_active":"2024-03-30T13:28:01.336784+0000","last_peered":"2024-03-30T13:28:01.336784+0000","last_clean":"2024-03-30T13:28:01.336784+0000","last_became_active":"2024-03-30T13:25:20.549084+0000","last_became_peered":"2024-03-30T13:25:20.549084+0000","last_unstale":"2024-03-30T13:28:01.336784+0000","last_undegraded":"2024-03-30T13:28:01.336784+0000","last_fullsized":"2024-03-30T13:28:01.336784+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:27:55.467102+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:27:55.467102+0000","last_clean_scrub_stamp":"2024-03-30T13:27:55.467102+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:10.379624+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":13764735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1925,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.066368+0000","last_change":"2024-03-30T13:27:26.798457+0000","last_active":"2024-03-30T13:28:01.066368+0000","last_peered":"2024-03-30T13:28:01.066368+0000","last_clean":"2024-03-30T13:28:01.066368+0000","last_became_active":"2024-03-30T13:26:16.583003+0000","last_became_peered":"2024-03-30T13:26:16.583003+0000","last_unstale":"2024-03-30T13:28:01.066368+0000","last_undegraded":"2024-03-30T13:28:01.066368+0000","last_fullsized":"2024-03-30T13:28:01.066368+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:26.798199+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:27:26.798199+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:54.470648+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":185,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T13:28:00.469884+0000","last_change":"2024-03-30T13:27:07.484434+0000","last_active":"2024-03-30T13:28:00.469884+0000","last_peered":"2024-03-30T13:28:00.469884+0000","last_clean":"2024-03-30T13:28:00.469884+0000","last_became_active":"2024-03-30T13:27:07.482993+0000","last_became_peered":"2024-03-30T13:27:07.482993+0000","last_unstale":"2024-03-30T13:28:00.469884+0000","last_undegraded":"2024-03-30T13:28:00.469884+0000","last_fullsized":"2024-03-30T13:28:00.469884+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:26:47.672328+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:26:47.672328+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:02.109948+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":171,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T13:28:00.469815+0000","last_change":"2024-03-30T13:27:48.638120+0000","last_active":"2024-03-30T13:28:00.469815+0000","last_peered":"2024-03-30T13:28:00.469815+0000","last_clean":"2024-03-30T13:28:00.469815+0000","last_became_active":"2024-03-30T13:26:16.577250+0000","last_became_peered":"2024-03-30T13:26:16.577250+0000","last_unstale":"2024-03-30T13:28:00.469815+0000","last_undegraded":"2024-03-30T13:28:00.469815+0000","last_fullsized":"2024-03-30T13:28:00.469815+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:27:48.637751+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:27:48.637751+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:15.915191+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":839,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.098817+0000","last_change":"2024-03-30T13:27:01.991450+0000","last_active":"2024-03-30T13:28:01.098817+0000","last_peered":"2024-03-30T13:28:01.098817+0000","last_clean":"2024-03-30T13:28:01.098817+0000","last_became_active":"2024-03-30T13:25:20.556165+0000","last_became_peered":"2024-03-30T13:25:20.556165+0000","last_unstale":"2024-03-30T13:28:01.098817+0000","last_undegraded":"2024-03-30T13:28:01.098817+0000","last_fullsized":"2024-03-30T13:28:01.098817+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:01.991062+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:27:01.991062+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:06.741931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1976,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.067440+0000","last_change":"2024-03-30T13:27:39.308222+0000","last_active":"2024-03-30T13:28:01.067440+0000","last_peered":"2024-03-30T13:28:01.067440+0000","last_clean":"2024-03-30T13:28:01.067440+0000","last_became_active":"2024-03-30T13:25:20.551941+0000","last_became_peered":"2024-03-30T13:25:20.551941+0000","last_unstale":"2024-03-30T13:28:01.067440+0000","last_undegraded":"2024-03-30T13:28:01.067440+0000","last_fullsized":"2024-03-30T13:28:01.067440+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:27:39.308113+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:27:39.308113+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:47.332129+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11394981,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":178,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T13:28:00.469821+0000","last_change":"2024-03-30T13:27:12.597085+0000","last_active":"2024-03-30T13:28:00.469821+0000","last_peered":"2024-03-30T13:28:00.469821+0000","last_clean":"2024-03-30T13:28:00.469821+0000","last_became_active":"2024-03-30T13:26:16.583861+0000","last_became_peered":"2024-03-30T13:26:16.583861+0000","last_unstale":"2024-03-30T13:28:00.469821+0000","last_undegraded":"2024-03-30T13:28:00.469821+0000","last_fullsized":"2024-03-30T13:28:00.469821+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:27:12.596825+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:27:12.596825+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:25.628559+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1284,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.126491+0000","last_change":"2024-03-30T13:27:22.661584+0000","last_active":"2024-03-30T13:28:01.126491+0000","last_peered":"2024-03-30T13:28:01.126491+0000","last_clean":"2024-03-30T13:28:01.126491+0000","last_became_active":"2024-03-30T13:27:07.598881+0000","last_became_peered":"2024-03-30T13:27:07.598881+0000","last_unstale":"2024-03-30T13:28:01.126491+0000","last_undegraded":"2024-03-30T13:28:01.126491+0000","last_fullsized":"2024-03-30T13:28:01.126491+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:27:22.661428+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:27:22.661428+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:51.172824+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":2097,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.067339+0000","last_change":"2024-03-30T13:27:41.019396+0000","last_active":"2024-03-30T13:28:01.067339+0000","last_peered":"2024-03-30T13:28:01.067339+0000","last_clean":"2024-03-30T13:28:01.067339+0000","last_became_active":"2024-03-30T13:26:16.582448+0000","last_became_peered":"2024-03-30T13:26:16.582448+0000","last_unstale":"2024-03-30T13:28:01.067339+0000","last_undegraded":"2024-03-30T13:28:01.067339+0000","last_fullsized":"2024-03-30T13:28:01.067339+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:27:41.018992+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:27:41.018992+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:06.568280+0000","scrub_duration":715,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":76134720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1735,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.066182+0000","last_change":"2024-03-30T13:27:58.853482+0000","last_active":"2024-03-30T13:28:01.066182+0000","last_peered":"2024-03-30T13:28:01.066182+0000","last_clean":"2024-03-30T13:28:01.066182+0000","last_became_active":"2024-03-30T13:24:29.442802+0000","last_became_peered":"2024-03-30T13:24:29.442802+0000","last_unstale":"2024-03-30T13:28:01.066182+0000","last_undegraded":"2024-03-30T13:28:01.066182+0000","last_fullsized":"2024-03-30T13:28:01.066182+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:58.853242+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:27:58.853242+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:12.971711+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":2044,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.336514+0000","last_change":"2024-03-30T13:27:17.528668+0000","last_active":"2024-03-30T13:28:01.336514+0000","last_peered":"2024-03-30T13:28:01.336514+0000","last_clean":"2024-03-30T13:28:01.336514+0000","last_became_active":"2024-03-30T13:24:29.447202+0000","last_became_peered":"2024-03-30T13:24:29.447202+0000","last_unstale":"2024-03-30T13:28:01.336514+0000","last_undegraded":"2024-03-30T13:28:01.336514+0000","last_fullsized":"2024-03-30T13:28:01.336514+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:27:17.528414+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:27:17.528414+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:25.562324+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":2138,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.336546+0000","last_change":"2024-03-30T13:27:28.612011+0000","last_active":"2024-03-30T13:28:01.336546+0000","last_peered":"2024-03-30T13:28:01.336546+0000","last_clean":"2024-03-30T13:28:01.336546+0000","last_became_active":"2024-03-30T13:25:20.549465+0000","last_became_peered":"2024-03-30T13:25:20.549465+0000","last_unstale":"2024-03-30T13:28:01.336546+0000","last_undegraded":"2024-03-30T13:28:01.336546+0000","last_fullsized":"2024-03-30T13:28:01.336546+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:27:28.611732+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:27:28.611732+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:50.237192+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":892,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.098817+0000","last_change":"2024-03-30T13:27:42.924031+0000","last_active":"2024-03-30T13:28:01.098817+0000","last_peered":"2024-03-30T13:28:01.098817+0000","last_clean":"2024-03-30T13:28:01.098817+0000","last_became_active":"2024-03-30T13:25:20.549427+0000","last_became_peered":"2024-03-30T13:25:20.549427+0000","last_unstale":"2024-03-30T13:28:01.098817+0000","last_undegraded":"2024-03-30T13:28:01.098817+0000","last_fullsized":"2024-03-30T13:28:01.098817+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:27:42.923778+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:27:42.923778+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:12.595140+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":169,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T13:28:00.469819+0000","last_change":"2024-03-30T13:27:51.677273+0000","last_active":"2024-03-30T13:28:00.469819+0000","last_peered":"2024-03-30T13:28:00.469819+0000","last_clean":"2024-03-30T13:28:00.469819+0000","last_became_active":"2024-03-30T13:26:16.578484+0000","last_became_peered":"2024-03-30T13:26:16.578484+0000","last_unstale":"2024-03-30T13:28:00.469819+0000","last_undegraded":"2024-03-30T13:28:00.469819+0000","last_fullsized":"2024-03-30T13:28:00.469819+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:51.677029+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:27:51.677029+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:14.237232+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":907,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.098821+0000","last_change":"2024-03-30T13:27:35.881662+0000","last_active":"2024-03-30T13:28:01.098821+0000","last_peered":"2024-03-30T13:28:01.098821+0000","last_clean":"2024-03-30T13:28:01.098821+0000","last_became_active":"2024-03-30T13:25:20.556235+0000","last_became_peered":"2024-03-30T13:25:20.556235+0000","last_unstale":"2024-03-30T13:28:01.098821+0000","last_undegraded":"2024-03-30T13:28:01.098821+0000","last_fullsized":"2024-03-30T13:28:01.098821+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:27:35.881272+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_clean_scrub_stamp":"2024-03-30T13:27:35.881272+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:48.834909+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":1147,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.126491+0000","last_change":"2024-03-30T13:27:30.772468+0000","last_active":"2024-03-30T13:28:01.126491+0000","last_peered":"2024-03-30T13:28:01.126491+0000","last_clean":"2024-03-30T13:28:01.126491+0000","last_became_active":"2024-03-30T13:27:07.478858+0000","last_became_peered":"2024-03-30T13:27:07.478858+0000","last_unstale":"2024-03-30T13:28:01.126491+0000","last_undegraded":"2024-03-30T13:28:01.126491+0000","last_fullsized":"2024-03-30T13:28:01.126491+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:27:30.772331+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:27:30.772331+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:38.830587+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":276723559,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387788,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60232,"kb_used_data":23036,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94311608,"statfs":{"total":96636764160,"available":96575086592,"internally_reserved":0,"allocated":23588864,"data_stored":19148634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":816,"seq":3504693313571,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69588,"kb_used_data":27780,"kb_used_omap":12,"kb_used_meta":41779,"kb_avail":94302252,"statfs":{"total":96636764160,"available":96565506048,"internally_reserved":0,"allocated":28446720,"data_stored":24006490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12654,"internal_metadata":42782354},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":794,"seq":3410204033070,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60012,"kb_used_data":19484,"kb_used_omap":15,"kb_used_meta":40496,"kb_avail":94311828,"statfs":{"total":96636764160,"available":96575311872,"internally_reserved":0,"allocated":19951616,"data_stored":15511386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15941,"internal_metadata":41468347},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509834,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53752,"kb_used_data":14252,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94318088,"statfs":{"total":96636764160,"available":96581722112,"internally_reserved":0,"allocated":14594048,"data_stored":10153818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":861,"seq":3697966841871,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51180,"kb_used_data":13528,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94320660,"statfs":{"total":96636764160,"available":96584355840,"internally_reserved":0,"allocated":13852672,"data_stored":9412442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12649,"internal_metadata":38522519},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509835,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56756,"kb_used_data":21416,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315084,"statfs":{"total":96636764160,"available":96578646016,"internally_reserved":0,"allocated":21929984,"data_stored":17496555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":837,"seq":3594887626777,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58996,"kb_used_data":17960,"kb_used_omap":17,"kb_used_meta":41006,"kb_avail":94312844,"statfs":{"total":96636764160,"available":96576352256,"internally_reserved":0,"allocated":18391040,"data_stored":13957611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17588,"internal_metadata":41990988},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570178,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57176,"kb_used_data":20428,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94314664,"statfs":{"total":96636764160,"available":96578215936,"internally_reserved":0,"allocated":20918272,"data_stored":16478042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:28:05.751 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-30T13:28:05.751 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:28:05.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:05.855+0000 7fab664c2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:05.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:05.867+0000 7fab664c2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:06.038 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:28:06.039 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:28:06.055 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1881,"stamp":"2024-03-30T13:28:04.349721+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":276723559,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":467692,"kb_used_data":157884,"kb_used_omap":77,"kb_used_meta":309618,"kb_avail":754507028,"statfs":{"total":773094113280,"available":772615196672,"internally_reserved":0,"allocated":161673216,"data_stored":126164978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":79062,"internal_metadata":317049642},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.313204"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":2037,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.336784+0000","last_change":"2024-03-30T13:27:55.467519+0000","last_active":"2024-03-30T13:28:01.336784+0000","last_peered":"2024-03-30T13:28:01.336784+0000","last_clean":"2024-03-30T13:28:01.336784+0000","last_became_active":"2024-03-30T13:25:20.549084+0000","last_became_peered":"2024-03-30T13:25:20.549084+0000","last_unstale":"2024-03-30T13:28:01.336784+0000","last_undegraded":"2024-03-30T13:28:01.336784+0000","last_fullsized":"2024-03-30T13:28:01.336784+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:27:55.467102+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:27:55.467102+0000","last_clean_scrub_stamp":"2024-03-30T13:27:55.467102+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:10.379624+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":13764735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1925,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.066368+0000","last_change":"2024-03-30T13:27:26.798457+0000","last_active":"2024-03-30T13:28:01.066368+0000","last_peered":"2024-03-30T13:28:01.066368+0000","last_clean":"2024-03-30T13:28:01.066368+0000","last_became_active":"2024-03-30T13:26:16.583003+0000","last_became_peered":"2024-03-30T13:26:16.583003+0000","last_unstale":"2024-03-30T13:28:01.066368+0000","last_undegraded":"2024-03-30T13:28:01.066368+0000","last_fullsized":"2024-03-30T13:28:01.066368+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:26.798199+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:27:26.798199+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:54.470648+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":185,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T13:28:00.469884+0000","last_change":"2024-03-30T13:27:07.484434+0000","last_active":"2024-03-30T13:28:00.469884+0000","last_peered":"2024-03-30T13:28:00.469884+0000","last_clean":"2024-03-30T13:28:00.469884+0000","last_became_active":"2024-03-30T13:27:07.482993+0000","last_became_peered":"2024-03-30T13:27:07.482993+0000","last_unstale":"2024-03-30T13:28:00.469884+0000","last_undegraded":"2024-03-30T13:28:00.469884+0000","last_fullsized":"2024-03-30T13:28:00.469884+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:26:47.672328+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:26:47.672328+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:02.109948+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":171,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T13:28:00.469815+0000","last_change":"2024-03-30T13:27:48.638120+0000","last_active":"2024-03-30T13:28:00.469815+0000","last_peered":"2024-03-30T13:28:00.469815+0000","last_clean":"2024-03-30T13:28:00.469815+0000","last_became_active":"2024-03-30T13:26:16.577250+0000","last_became_peered":"2024-03-30T13:26:16.577250+0000","last_unstale":"2024-03-30T13:28:00.469815+0000","last_undegraded":"2024-03-30T13:28:00.469815+0000","last_fullsized":"2024-03-30T13:28:00.469815+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:27:48.637751+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:27:48.637751+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:15.915191+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":839,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.098817+0000","last_change":"2024-03-30T13:27:01.991450+0000","last_active":"2024-03-30T13:28:01.098817+0000","last_peered":"2024-03-30T13:28:01.098817+0000","last_clean":"2024-03-30T13:28:01.098817+0000","last_became_active":"2024-03-30T13:25:20.556165+0000","last_became_peered":"2024-03-30T13:25:20.556165+0000","last_unstale":"2024-03-30T13:28:01.098817+0000","last_undegraded":"2024-03-30T13:28:01.098817+0000","last_fullsized":"2024-03-30T13:28:01.098817+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:01.991062+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:27:01.991062+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:06.741931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1976,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.067440+0000","last_change":"2024-03-30T13:27:39.308222+0000","last_active":"2024-03-30T13:28:01.067440+0000","last_peered":"2024-03-30T13:28:01.067440+0000","last_clean":"2024-03-30T13:28:01.067440+0000","last_became_active":"2024-03-30T13:25:20.551941+0000","last_became_peered":"2024-03-30T13:25:20.551941+0000","last_unstale":"2024-03-30T13:28:01.067440+0000","last_undegraded":"2024-03-30T13:28:01.067440+0000","last_fullsized":"2024-03-30T13:28:01.067440+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:27:39.308113+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:27:39.308113+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:47.332129+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11394981,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":178,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T13:28:00.469821+0000","last_change":"2024-03-30T13:27:12.597085+0000","last_active":"2024-03-30T13:28:00.469821+0000","last_peered":"2024-03-30T13:28:00.469821+0000","last_clean":"2024-03-30T13:28:00.469821+0000","last_became_active":"2024-03-30T13:26:16.583861+0000","last_became_peered":"2024-03-30T13:26:16.583861+0000","last_unstale":"2024-03-30T13:28:00.469821+0000","last_undegraded":"2024-03-30T13:28:00.469821+0000","last_fullsized":"2024-03-30T13:28:00.469821+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:27:12.596825+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:27:12.596825+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:25.628559+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1284,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.126491+0000","last_change":"2024-03-30T13:27:22.661584+0000","last_active":"2024-03-30T13:28:01.126491+0000","last_peered":"2024-03-30T13:28:01.126491+0000","last_clean":"2024-03-30T13:28:01.126491+0000","last_became_active":"2024-03-30T13:27:07.598881+0000","last_became_peered":"2024-03-30T13:27:07.598881+0000","last_unstale":"2024-03-30T13:28:01.126491+0000","last_undegraded":"2024-03-30T13:28:01.126491+0000","last_fullsized":"2024-03-30T13:28:01.126491+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:27:22.661428+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:27:22.661428+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:51.172824+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":2097,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.067339+0000","last_change":"2024-03-30T13:27:41.019396+0000","last_active":"2024-03-30T13:28:01.067339+0000","last_peered":"2024-03-30T13:28:01.067339+0000","last_clean":"2024-03-30T13:28:01.067339+0000","last_became_active":"2024-03-30T13:26:16.582448+0000","last_became_peered":"2024-03-30T13:26:16.582448+0000","last_unstale":"2024-03-30T13:28:01.067339+0000","last_undegraded":"2024-03-30T13:28:01.067339+0000","last_fullsized":"2024-03-30T13:28:01.067339+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:27:41.018992+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:27:41.018992+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:06.568280+0000","scrub_duration":715,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":76134720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1735,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.066182+0000","last_change":"2024-03-30T13:27:58.853482+0000","last_active":"2024-03-30T13:28:01.066182+0000","last_peered":"2024-03-30T13:28:01.066182+0000","last_clean":"2024-03-30T13:28:01.066182+0000","last_became_active":"2024-03-30T13:24:29.442802+0000","last_became_peered":"2024-03-30T13:24:29.442802+0000","last_unstale":"2024-03-30T13:28:01.066182+0000","last_undegraded":"2024-03-30T13:28:01.066182+0000","last_fullsized":"2024-03-30T13:28:01.066182+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:58.853242+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:27:58.853242+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:12.971711+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":2044,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.336514+0000","last_change":"2024-03-30T13:27:17.528668+0000","last_active":"2024-03-30T13:28:01.336514+0000","last_peered":"2024-03-30T13:28:01.336514+0000","last_clean":"2024-03-30T13:28:01.336514+0000","last_became_active":"2024-03-30T13:24:29.447202+0000","last_became_peered":"2024-03-30T13:24:29.447202+0000","last_unstale":"2024-03-30T13:28:01.336514+0000","last_undegraded":"2024-03-30T13:28:01.336514+0000","last_fullsized":"2024-03-30T13:28:01.336514+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:27:17.528414+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:27:17.528414+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:25.562324+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":2138,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.336546+0000","last_change":"2024-03-30T13:27:28.612011+0000","last_active":"2024-03-30T13:28:01.336546+0000","last_peered":"2024-03-30T13:28:01.336546+0000","last_clean":"2024-03-30T13:28:01.336546+0000","last_became_active":"2024-03-30T13:25:20.549465+0000","last_became_peered":"2024-03-30T13:25:20.549465+0000","last_unstale":"2024-03-30T13:28:01.336546+0000","last_undegraded":"2024-03-30T13:28:01.336546+0000","last_fullsized":"2024-03-30T13:28:01.336546+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:27:28.611732+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:27:28.611732+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:50.237192+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":892,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.098817+0000","last_change":"2024-03-30T13:27:42.924031+0000","last_active":"2024-03-30T13:28:01.098817+0000","last_peered":"2024-03-30T13:28:01.098817+0000","last_clean":"2024-03-30T13:28:01.098817+0000","last_became_active":"2024-03-30T13:25:20.549427+0000","last_became_peered":"2024-03-30T13:25:20.549427+0000","last_unstale":"2024-03-30T13:28:01.098817+0000","last_undegraded":"2024-03-30T13:28:01.098817+0000","last_fullsized":"2024-03-30T13:28:01.098817+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:27:42.923778+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:27:42.923778+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:12.595140+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":169,"reported_epoch":910,"state":"active+clean","last_fresh":"2024-03-30T13:28:00.469819+0000","last_change":"2024-03-30T13:27:51.677273+0000","last_active":"2024-03-30T13:28:00.469819+0000","last_peered":"2024-03-30T13:28:00.469819+0000","last_clean":"2024-03-30T13:28:00.469819+0000","last_became_active":"2024-03-30T13:26:16.578484+0000","last_became_peered":"2024-03-30T13:26:16.578484+0000","last_unstale":"2024-03-30T13:28:00.469819+0000","last_undegraded":"2024-03-30T13:28:00.469819+0000","last_fullsized":"2024-03-30T13:28:00.469819+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:51.677029+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:27:51.677029+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:14.237232+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":907,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.098821+0000","last_change":"2024-03-30T13:27:35.881662+0000","last_active":"2024-03-30T13:28:01.098821+0000","last_peered":"2024-03-30T13:28:01.098821+0000","last_clean":"2024-03-30T13:28:01.098821+0000","last_became_active":"2024-03-30T13:25:20.556235+0000","last_became_peered":"2024-03-30T13:25:20.556235+0000","last_unstale":"2024-03-30T13:28:01.098821+0000","last_undegraded":"2024-03-30T13:28:01.098821+0000","last_fullsized":"2024-03-30T13:28:01.098821+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:27:35.881272+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_clean_scrub_stamp":"2024-03-30T13:27:35.881272+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:48.834909+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":1147,"reported_epoch":911,"state":"active+clean","last_fresh":"2024-03-30T13:28:01.126491+0000","last_change":"2024-03-30T13:27:30.772468+0000","last_active":"2024-03-30T13:28:01.126491+0000","last_peered":"2024-03-30T13:28:01.126491+0000","last_clean":"2024-03-30T13:28:01.126491+0000","last_became_active":"2024-03-30T13:27:07.478858+0000","last_became_peered":"2024-03-30T13:27:07.478858+0000","last_unstale":"2024-03-30T13:28:01.126491+0000","last_undegraded":"2024-03-30T13:28:01.126491+0000","last_fullsized":"2024-03-30T13:28:01.126491+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:27:30.772331+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:27:30.772331+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:38.830587+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":276723559,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387788,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60232,"kb_used_data":23036,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94311608,"statfs":{"total":96636764160,"available":96575086592,"internally_reserved":0,"allocated":23588864,"data_stored":19148634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":816,"seq":3504693313571,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69588,"kb_used_data":27780,"kb_used_omap":12,"kb_used_meta":41779,"kb_avail":94302252,"statfs":{"total":96636764160,"available":96565506048,"internally_reserved":0,"allocated":28446720,"data_stored":24006490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12654,"internal_metadata":42782354},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":794,"seq":3410204033070,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60012,"kb_used_data":19484,"kb_used_omap":15,"kb_used_meta":40496,"kb_avail":94311828,"statfs":{"total":96636764160,"available":96575311872,"internally_reserved":0,"allocated":19951616,"data_stored":15511386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15941,"internal_metadata":41468347},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509834,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53752,"kb_used_data":14252,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94318088,"statfs":{"total":96636764160,"available":96581722112,"internally_reserved":0,"allocated":14594048,"data_stored":10153818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":861,"seq":3697966841871,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51180,"kb_used_data":13528,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94320660,"statfs":{"total":96636764160,"available":96584355840,"internally_reserved":0,"allocated":13852672,"data_stored":9412442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12649,"internal_metadata":38522519},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509835,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56756,"kb_used_data":21416,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315084,"statfs":{"total":96636764160,"available":96578646016,"internally_reserved":0,"allocated":21929984,"data_stored":17496555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":837,"seq":3594887626777,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58996,"kb_used_data":17960,"kb_used_omap":17,"kb_used_meta":41006,"kb_avail":94312844,"statfs":{"total":96636764160,"available":96576352256,"internally_reserved":0,"allocated":18391040,"data_stored":13957611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17588,"internal_metadata":41990988},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570178,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57176,"kb_used_data":20428,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94314664,"statfs":{"total":96636764160,"available":96578215936,"internally_reserved":0,"allocated":20918272,"data_stored":16478042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:28:06.056 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-30T13:28:06.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:06.159+0000 7f54a16be640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:06.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:06.171+0000 7f54a16be640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:06.340 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:28:06.341 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":912,"fsid":"0dfa8b7f-0f61-47f0-a4fe-4442c2d72a0a","created":"2024-03-30T12:55:22.904201+0000","modified":"2024-03-30T13:28:04.079697+0000","last_up_change":"2024-03-30T13:27:06.453839+0000","last_in_change":"2024-03-30T13:27:06.453839+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":42,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"unique_pool_0","create_time":"2024-03-30T12:55:41.942458+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":23,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1320388d-ee71-45f3-8a07-cb3f497e9c28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":787,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6809","nonce":2813590964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6811","nonce":2813590964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6815","nonce":2813590964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2813590964},{"type":"v1","addr":"172.21.15.67:6813","nonce":2813590964}]},"public_addr":"172.21.15.67:6809/2813590964","cluster_addr":"172.21.15.67:6811/2813590964","heartbeat_back_addr":"172.21.15.67:6815/2813590964","heartbeat_front_addr":"172.21.15.67:6813/2813590964","state":["exists","up"]},{"osd":1,"uuid":"0cd00c37-773f-4a6e-bb54-deed4a0d2a61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":769,"last_clean_end":854,"up_from":861,"up_thru":861,"down_at":855,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2045515420},{"type":"v1","addr":"172.21.15.67:6818","nonce":2045515420}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2045515420},{"type":"v1","addr":"172.21.15.67:6821","nonce":2045515420}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2045515420},{"type":"v1","addr":"172.21.15.67:6828","nonce":2045515420}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6823","nonce":2045515420},{"type":"v1","addr":"172.21.15.67:6825","nonce":2045515420}]},"public_addr":"172.21.15.67:6818/2045515420","cluster_addr":"172.21.15.67:6821/2045515420","heartbeat_back_addr":"172.21.15.67:6828/2045515420","heartbeat_front_addr":"172.21.15.67:6825/2045515420","state":["exists","up"]},{"osd":2,"uuid":"0be6d4c0-5d83-485d-afb7-461810f7efbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":855,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6817","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6819","nonce":3696364776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6824","nonce":3696364776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6831","nonce":3696364776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6827","nonce":3696364776},{"type":"v1","addr":"172.21.15.67:6829","nonce":3696364776}]},"public_addr":"172.21.15.67:6819/3696364776","cluster_addr":"172.21.15.67:6824/3696364776","heartbeat_back_addr":"172.21.15.67:6831/3696364776","heartbeat_front_addr":"172.21.15.67:6829/3696364776","state":["exists","up"]},{"osd":3,"uuid":"f851bae3-8485-4e67-8c62-7c9eca784b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":699,"last_clean_end":738,"up_from":747,"up_thru":838,"down_at":739,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6801","nonce":3161027348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6803","nonce":3161027348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6807","nonce":3161027348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3161027348},{"type":"v1","addr":"172.21.15.67:6805","nonce":3161027348}]},"public_addr":"172.21.15.67:6801/3161027348","cluster_addr":"172.21.15.67:6803/3161027348","heartbeat_back_addr":"172.21.15.67:6807/3161027348","heartbeat_front_addr":"172.21.15.67:6805/3161027348","state":["exists","up"]},{"osd":4,"uuid":"515f79f0-fb30-4438-9668-c32ce26a63fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":677,"last_clean_end":785,"up_from":794,"up_thru":855,"down_at":786,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":167623373},{"type":"v1","addr":"172.21.15.82:6803","nonce":167623373}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6805","nonce":167623373},{"type":"v1","addr":"172.21.15.82:6807","nonce":167623373}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6813","nonce":167623373},{"type":"v1","addr":"172.21.15.82:6815","nonce":167623373}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6809","nonce":167623373},{"type":"v1","addr":"172.21.15.82:6811","nonce":167623373}]},"public_addr":"172.21.15.82:6803/167623373","cluster_addr":"172.21.15.82:6807/167623373","heartbeat_back_addr":"172.21.15.82:6815/167623373","heartbeat_front_addr":"172.21.15.82:6811/167623373","state":["exists","up"]},{"osd":5,"uuid":"e73cd117-12e6-47b5-8561-e5ede18b977c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":515,"last_clean_end":810,"up_from":816,"up_thru":833,"down_at":811,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":1480534728},{"type":"v1","addr":"172.21.15.82:6817","nonce":1480534728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":1480534728},{"type":"v1","addr":"172.21.15.82:6819","nonce":1480534728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":1480534728},{"type":"v1","addr":"172.21.15.82:6823","nonce":1480534728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":1480534728},{"type":"v1","addr":"172.21.15.82:6821","nonce":1480534728}]},"public_addr":"172.21.15.82:6817/1480534728","cluster_addr":"172.21.15.82:6819/1480534728","heartbeat_back_addr":"172.21.15.82:6823/1480534728","heartbeat_front_addr":"172.21.15.82:6821/1480534728","state":["exists","up"]},{"osd":6,"uuid":"9cb2aa10-05de-4b9c-bd43-56773f909acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":629,"last_clean_end":715,"up_from":722,"up_thru":816,"down_at":716,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6801","nonce":326680821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6806","nonce":326680821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6814","nonce":326680821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":326680821},{"type":"v1","addr":"172.21.15.82:6810","nonce":326680821}]},"public_addr":"172.21.15.82:6801/326680821","cluster_addr":"172.21.15.82:6806/326680821","heartbeat_back_addr":"172.21.15.82:6814/326680821","heartbeat_front_addr":"172.21.15.82:6810/326680821","state":["exists","up"]},{"osd":7,"uuid":"cc3bfce6-25bf-4f70-8f70-ee26ea758ba1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":650,"last_clean_end":832,"up_from":837,"up_thru":861,"down_at":833,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":876600018},{"type":"v1","addr":"172.21.15.82:6825","nonce":876600018}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":876600018},{"type":"v1","addr":"172.21.15.82:6827","nonce":876600018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":876600018},{"type":"v1","addr":"172.21.15.82:6831","nonce":876600018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":876600018},{"type":"v1","addr":"172.21.15.82:6829","nonce":876600018}]},"public_addr":"172.21.15.82:6825/876600018","cluster_addr":"172.21.15.82:6827/876600018","heartbeat_back_addr":"172.21.15.82:6831/876600018","heartbeat_front_addr":"172.21.15.82:6829/876600018","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:28:03.997375+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:27:57.710645+0000","dead_epoch":855},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:27:41.623030+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:26:47.427678+0000","dead_epoch":739},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:27:20.230932+0000","dead_epoch":786},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:27:40.166425+0000","dead_epoch":811},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:27:11.815072+0000","dead_epoch":716},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-30T13:26:48.975089+0000","dead_epoch":833}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-30T13:28:07.360 INFO:tasks.ceph:Scrubbing osd.0 2024-03-30T13:28:07.360 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-30T13:28:07.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:07.483+0000 7f39db772640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:07.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:07.503+0000 7f39db772640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:07.530 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-30T13:28:07.530 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-30T13:28:07.530 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-30T13:28:07.547 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-30T13:28:07.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:07.655+0000 7fdb1c0fe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:07.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:07.667+0000 7fdb1c0fe640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:07.841 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 0 to deep-scrub 2024-03-30T13:28:07.860 INFO:tasks.ceph:Scrubbing osd.1 2024-03-30T13:28:07.860 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-30T13:28:07.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:07.963+0000 7f47f5368640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:07.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:07.983+0000 7f47f5368640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:08.008 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-30T13:28:08.008 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-30T13:28:08.008 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-30T13:28:08.024 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-30T13:28:08.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:08.127+0000 7effd22a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:08.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:08.143+0000 7effd22a6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:08.314 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 1 to deep-scrub 2024-03-30T13:28:08.333 INFO:tasks.ceph:Scrubbing osd.2 2024-03-30T13:28:08.333 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-30T13:28:08.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:08.435+0000 7fe200fda640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:08.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:08.455+0000 7fe200fda640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:08.481 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-30T13:28:08.481 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-30T13:28:08.482 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-30T13:28:08.496 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-30T13:28:08.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:08.591+0000 7f3d14e17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:08.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:08.607+0000 7f3d14e17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:08.771 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 2 to deep-scrub 2024-03-30T13:28:08.792 INFO:tasks.ceph:Scrubbing osd.3 2024-03-30T13:28:08.792 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-30T13:28:08.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:08.891+0000 7f2a62893640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:08.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:08.911+0000 7f2a62893640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:08.937 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-30T13:28:08.937 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-30T13:28:08.937 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-30T13:28:08.952 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-30T13:28:09.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:09.051+0000 7fdbd391d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:09.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:09.063+0000 7fdbd391d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:09.239 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 3 to deep-scrub 2024-03-30T13:28:09.257 INFO:tasks.ceph:Scrubbing osd.4 2024-03-30T13:28:09.257 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-30T13:28:09.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:09.371+0000 7f4fad239640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:09.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:09.387+0000 7f4fad239640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:09.419 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-30T13:28:09.419 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-03-30T13:28:09.419 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-30T13:28:09.435 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-30T13:28:09.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:09.543+0000 7fd3a03bc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:09.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:09.555+0000 7fd3a03bc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:09.731 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 4 to deep-scrub 2024-03-30T13:28:09.749 INFO:tasks.ceph:Scrubbing osd.5 2024-03-30T13:28:09.750 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-30T13:28:09.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:09.855+0000 7f5c1171c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:09.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:09.875+0000 7f5c1171c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:09.903 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-30T13:28:09.903 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-30T13:28:09.904 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-30T13:28:09.919 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-30T13:28:10.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:10.027+0000 7f2176392640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:10.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:10.043+0000 7f2176392640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:10.220 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 5 to deep-scrub 2024-03-30T13:28:10.239 INFO:tasks.ceph:Scrubbing osd.6 2024-03-30T13:28:10.239 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-30T13:28:10.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:10.355+0000 7fcfb1ac0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:10.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:10.371+0000 7fcfb1ac0640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:10.402 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-30T13:28:10.403 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-30T13:28:10.403 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-30T13:28:10.418 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-30T13:28:10.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:10.523+0000 7fd63d373640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:10.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:10.539+0000 7fd63d373640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:10.713 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 6 to deep-scrub 2024-03-30T13:28:10.732 INFO:tasks.ceph:Scrubbing osd.7 2024-03-30T13:28:10.733 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-30T13:28:10.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:10.843+0000 7f816db22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:10.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:10.867+0000 7f816db22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:10.897 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-03-30T13:28:10.897 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-30T13:28:10.898 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-03-30T13:28:10.915 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-30T13:28:11.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:11.023+0000 7f84e5a89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:11.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:11.039+0000 7f84e5a89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:11.257 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 7 to deep-scrub 2024-03-30T13:28:11.278 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:28:11.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:11.383+0000 7f892a00a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:11.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:11.395+0000 7f892a00a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:11.564 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:28:11.565 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:28:11.581 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1886,"stamp":"2024-03-30T13:28:10.351351+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":276723559,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":467880,"kb_used_data":158076,"kb_used_omap":77,"kb_used_meta":309618,"kb_avail":754506840,"statfs":{"total":773094113280,"available":772615004160,"internally_reserved":0,"allocated":161869824,"data_stored":126253358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":79062,"internal_metadata":317049642},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.269283"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":2039,"reported_epoch":912,"state":"active+clean","last_fresh":"2024-03-30T13:28:04.549050+0000","last_change":"2024-03-30T13:27:55.467519+0000","last_active":"2024-03-30T13:28:04.549050+0000","last_peered":"2024-03-30T13:28:04.549050+0000","last_clean":"2024-03-30T13:28:04.549050+0000","last_became_active":"2024-03-30T13:25:20.549084+0000","last_became_peered":"2024-03-30T13:25:20.549084+0000","last_unstale":"2024-03-30T13:28:04.549050+0000","last_undegraded":"2024-03-30T13:28:04.549050+0000","last_fullsized":"2024-03-30T13:28:04.549050+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:27:55.467102+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:27:55.467102+0000","last_clean_scrub_stamp":"2024-03-30T13:27:55.467102+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:10.379624+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":13764735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1931,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-03-30T13:28:08.113706+0000","last_change":"2024-03-30T13:27:26.798457+0000","last_active":"2024-03-30T13:28:08.113706+0000","last_peered":"2024-03-30T13:28:08.113706+0000","last_clean":"2024-03-30T13:28:08.113706+0000","last_became_active":"2024-03-30T13:26:16.583003+0000","last_became_peered":"2024-03-30T13:26:16.583003+0000","last_unstale":"2024-03-30T13:28:08.113706+0000","last_undegraded":"2024-03-30T13:28:08.113706+0000","last_fullsized":"2024-03-30T13:28:08.113706+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:26.798199+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:14:50.629162+0000","last_clean_scrub_stamp":"2024-03-30T13:27:26.798199+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:54.470648+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":197,"reported_epoch":912,"state":"active+clean","last_fresh":"2024-03-30T13:28:04.549823+0000","last_change":"2024-03-30T13:28:03.597972+0000","last_active":"2024-03-30T13:28:04.549823+0000","last_peered":"2024-03-30T13:28:04.549823+0000","last_clean":"2024-03-30T13:28:04.549823+0000","last_became_active":"2024-03-30T13:27:07.482993+0000","last_became_peered":"2024-03-30T13:27:07.482993+0000","last_unstale":"2024-03-30T13:28:04.549823+0000","last_undegraded":"2024-03-30T13:28:04.549823+0000","last_fullsized":"2024-03-30T13:28:04.549823+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:28:03.597587+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:22:07.121601+0000","last_clean_scrub_stamp":"2024-03-30T13:28:03.597587+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:16.416987+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":175,"reported_epoch":912,"state":"active+clean","last_fresh":"2024-03-30T13:28:04.549733+0000","last_change":"2024-03-30T13:27:48.638120+0000","last_active":"2024-03-30T13:28:04.549733+0000","last_peered":"2024-03-30T13:28:04.549733+0000","last_clean":"2024-03-30T13:28:04.549733+0000","last_became_active":"2024-03-30T13:26:16.577250+0000","last_became_peered":"2024-03-30T13:26:16.577250+0000","last_unstale":"2024-03-30T13:28:04.549733+0000","last_undegraded":"2024-03-30T13:28:04.549733+0000","last_fullsized":"2024-03-30T13:28:04.549733+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:27:48.637751+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:36.180557+0000","last_clean_scrub_stamp":"2024-03-30T13:27:48.637751+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:15.915191+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":853,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-03-30T13:28:08.500707+0000","last_change":"2024-03-30T13:28:06.928075+0000","last_active":"2024-03-30T13:28:08.500707+0000","last_peered":"2024-03-30T13:28:08.500707+0000","last_clean":"2024-03-30T13:28:08.500707+0000","last_became_active":"2024-03-30T13:25:20.556165+0000","last_became_peered":"2024-03-30T13:25:20.556165+0000","last_unstale":"2024-03-30T13:28:08.500707+0000","last_undegraded":"2024-03-30T13:28:08.500707+0000","last_fullsized":"2024-03-30T13:28:08.500707+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:28:06.927795+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:25.690651+0000","last_clean_scrub_stamp":"2024-03-30T13:28:06.927795+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:33.987884+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1978,"reported_epoch":912,"state":"active+clean","last_fresh":"2024-03-30T13:28:04.084236+0000","last_change":"2024-03-30T13:27:39.308222+0000","last_active":"2024-03-30T13:28:04.084236+0000","last_peered":"2024-03-30T13:28:04.084236+0000","last_clean":"2024-03-30T13:28:04.084236+0000","last_became_active":"2024-03-30T13:25:20.551941+0000","last_became_peered":"2024-03-30T13:25:20.551941+0000","last_unstale":"2024-03-30T13:28:04.084236+0000","last_undegraded":"2024-03-30T13:28:04.084236+0000","last_fullsized":"2024-03-30T13:28:04.084236+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:27:39.308113+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:21:21.676444+0000","last_clean_scrub_stamp":"2024-03-30T13:27:39.308113+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:47.332129+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11394981,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":182,"reported_epoch":912,"state":"active+clean","last_fresh":"2024-03-30T13:28:04.549810+0000","last_change":"2024-03-30T13:27:12.597085+0000","last_active":"2024-03-30T13:28:04.549810+0000","last_peered":"2024-03-30T13:28:04.549810+0000","last_clean":"2024-03-30T13:28:04.549810+0000","last_became_active":"2024-03-30T13:26:16.583861+0000","last_became_peered":"2024-03-30T13:26:16.583861+0000","last_unstale":"2024-03-30T13:28:04.549810+0000","last_undegraded":"2024-03-30T13:28:04.549810+0000","last_fullsized":"2024-03-30T13:28:04.549810+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:27:12.596825+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:00:34.155844+0000","last_clean_scrub_stamp":"2024-03-30T13:27:12.596825+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:25.628559+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1286,"reported_epoch":912,"state":"active+clean","last_fresh":"2024-03-30T13:28:04.548287+0000","last_change":"2024-03-30T13:27:22.661584+0000","last_active":"2024-03-30T13:28:04.548287+0000","last_peered":"2024-03-30T13:28:04.548287+0000","last_clean":"2024-03-30T13:28:04.548287+0000","last_became_active":"2024-03-30T13:27:07.598881+0000","last_became_peered":"2024-03-30T13:27:07.598881+0000","last_unstale":"2024-03-30T13:28:04.548287+0000","last_undegraded":"2024-03-30T13:28:04.548287+0000","last_fullsized":"2024-03-30T13:28:04.548287+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:27:22.661428+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:05:24.040489+0000","last_clean_scrub_stamp":"2024-03-30T13:27:22.661428+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:51.172824+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":2099,"reported_epoch":912,"state":"active+clean","last_fresh":"2024-03-30T13:28:04.084116+0000","last_change":"2024-03-30T13:27:41.019396+0000","last_active":"2024-03-30T13:28:04.084116+0000","last_peered":"2024-03-30T13:28:04.084116+0000","last_clean":"2024-03-30T13:28:04.084116+0000","last_became_active":"2024-03-30T13:26:16.582448+0000","last_became_peered":"2024-03-30T13:26:16.582448+0000","last_unstale":"2024-03-30T13:28:04.084116+0000","last_undegraded":"2024-03-30T13:28:04.084116+0000","last_fullsized":"2024-03-30T13:28:04.084116+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:27:41.018992+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:23:27.680784+0000","last_clean_scrub_stamp":"2024-03-30T13:27:41.018992+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:06.568280+0000","scrub_duration":715,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":76134720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1741,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-03-30T13:28:08.113583+0000","last_change":"2024-03-30T13:27:58.853482+0000","last_active":"2024-03-30T13:28:08.113583+0000","last_peered":"2024-03-30T13:28:08.113583+0000","last_clean":"2024-03-30T13:28:08.113583+0000","last_became_active":"2024-03-30T13:24:29.442802+0000","last_became_peered":"2024-03-30T13:24:29.442802+0000","last_unstale":"2024-03-30T13:28:08.113583+0000","last_undegraded":"2024-03-30T13:28:08.113583+0000","last_fullsized":"2024-03-30T13:28:08.113583+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:58.853242+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:20:23.631067+0000","last_clean_scrub_stamp":"2024-03-30T13:27:58.853242+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:12.971711+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":2046,"reported_epoch":912,"state":"active+clean","last_fresh":"2024-03-30T13:28:04.549000+0000","last_change":"2024-03-30T13:27:17.528668+0000","last_active":"2024-03-30T13:28:04.549000+0000","last_peered":"2024-03-30T13:28:04.549000+0000","last_clean":"2024-03-30T13:28:04.549000+0000","last_became_active":"2024-03-30T13:24:29.447202+0000","last_became_peered":"2024-03-30T13:24:29.447202+0000","last_unstale":"2024-03-30T13:28:04.549000+0000","last_undegraded":"2024-03-30T13:28:04.549000+0000","last_fullsized":"2024-03-30T13:28:04.549000+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:27:17.528414+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:20:42.398509+0000","last_clean_scrub_stamp":"2024-03-30T13:27:17.528414+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:25.562324+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":2140,"reported_epoch":912,"state":"active+clean","last_fresh":"2024-03-30T13:28:04.548745+0000","last_change":"2024-03-30T13:27:28.612011+0000","last_active":"2024-03-30T13:28:04.548745+0000","last_peered":"2024-03-30T13:28:04.548745+0000","last_clean":"2024-03-30T13:28:04.548745+0000","last_became_active":"2024-03-30T13:25:20.549465+0000","last_became_peered":"2024-03-30T13:25:20.549465+0000","last_unstale":"2024-03-30T13:28:04.548745+0000","last_undegraded":"2024-03-30T13:28:04.548745+0000","last_fullsized":"2024-03-30T13:28:04.548745+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:27:28.611732+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:23:22.573400+0000","last_clean_scrub_stamp":"2024-03-30T13:27:28.611732+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:50.237192+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":898,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-03-30T13:28:08.500622+0000","last_change":"2024-03-30T13:27:42.924031+0000","last_active":"2024-03-30T13:28:08.500622+0000","last_peered":"2024-03-30T13:28:08.500622+0000","last_clean":"2024-03-30T13:28:08.500622+0000","last_became_active":"2024-03-30T13:25:20.549427+0000","last_became_peered":"2024-03-30T13:25:20.549427+0000","last_unstale":"2024-03-30T13:28:08.500622+0000","last_undegraded":"2024-03-30T13:28:08.500622+0000","last_fullsized":"2024-03-30T13:28:08.500622+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:27:42.923778+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:07:53.685778+0000","last_clean_scrub_stamp":"2024-03-30T13:27:42.923778+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:12.595140+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":173,"reported_epoch":912,"state":"active+clean","last_fresh":"2024-03-30T13:28:04.549763+0000","last_change":"2024-03-30T13:27:51.677273+0000","last_active":"2024-03-30T13:28:04.549763+0000","last_peered":"2024-03-30T13:28:04.549763+0000","last_clean":"2024-03-30T13:28:04.549763+0000","last_became_active":"2024-03-30T13:26:16.578484+0000","last_became_peered":"2024-03-30T13:26:16.578484+0000","last_unstale":"2024-03-30T13:28:04.549763+0000","last_undegraded":"2024-03-30T13:28:04.549763+0000","last_fullsized":"2024-03-30T13:28:04.549763+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:27:51.677029+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T12:59:24.867416+0000","last_clean_scrub_stamp":"2024-03-30T13:27:51.677029+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:14.237232+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":913,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-03-30T13:28:08.500637+0000","last_change":"2024-03-30T13:27:35.881662+0000","last_active":"2024-03-30T13:28:08.500637+0000","last_peered":"2024-03-30T13:28:08.500637+0000","last_clean":"2024-03-30T13:28:08.500637+0000","last_became_active":"2024-03-30T13:25:20.556235+0000","last_became_peered":"2024-03-30T13:25:20.556235+0000","last_unstale":"2024-03-30T13:28:08.500637+0000","last_undegraded":"2024-03-30T13:28:08.500637+0000","last_fullsized":"2024-03-30T13:28:08.500637+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:27:35.881272+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:24:40.633617+0000","last_clean_scrub_stamp":"2024-03-30T13:27:35.881272+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:48.834909+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":1149,"reported_epoch":912,"state":"active+clean","last_fresh":"2024-03-30T13:28:04.548307+0000","last_change":"2024-03-30T13:27:30.772468+0000","last_active":"2024-03-30T13:28:04.548307+0000","last_peered":"2024-03-30T13:28:04.548307+0000","last_clean":"2024-03-30T13:28:04.548307+0000","last_became_active":"2024-03-30T13:27:07.478858+0000","last_became_peered":"2024-03-30T13:27:07.478858+0000","last_unstale":"2024-03-30T13:28:04.548307+0000","last_undegraded":"2024-03-30T13:28:04.548307+0000","last_fullsized":"2024-03-30T13:28:04.548307+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:27:30.772331+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:17:58.850996+0000","last_clean_scrub_stamp":"2024-03-30T13:27:30.772331+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:28:38.830587+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":276723559,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387789,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60244,"kb_used_data":23048,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94311596,"statfs":{"total":96636764160,"available":96575074304,"internally_reserved":0,"allocated":23601152,"data_stored":19154163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":816,"seq":3504693313573,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69624,"kb_used_data":27816,"kb_used_omap":12,"kb_used_meta":41779,"kb_avail":94302216,"statfs":{"total":96636764160,"available":96565469184,"internally_reserved":0,"allocated":28483584,"data_stored":24023077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12654,"internal_metadata":42782354},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":794,"seq":3410204033071,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60020,"kb_used_data":19496,"kb_used_omap":15,"kb_used_meta":40496,"kb_avail":94311820,"statfs":{"total":96636764160,"available":96575303680,"internally_reserved":0,"allocated":19963904,"data_stored":15516915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15941,"internal_metadata":41468347},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509836,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53792,"kb_used_data":14288,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94318048,"statfs":{"total":96636764160,"available":96581681152,"internally_reserved":0,"allocated":14630912,"data_stored":10170405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":861,"seq":3697966841872,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51184,"kb_used_data":13540,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94320656,"statfs":{"total":96636764160,"available":96584351744,"internally_reserved":0,"allocated":13864960,"data_stored":9417971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12649,"internal_metadata":38522519},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509837,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56808,"kb_used_data":21464,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315032,"statfs":{"total":96636764160,"available":96578592768,"internally_reserved":0,"allocated":21979136,"data_stored":17518629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":837,"seq":3594887626778,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59020,"kb_used_data":17984,"kb_used_omap":17,"kb_used_meta":41006,"kb_avail":94312820,"statfs":{"total":96636764160,"available":96576327680,"internally_reserved":0,"allocated":18415616,"data_stored":13968627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17588,"internal_metadata":41990988},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570179,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57188,"kb_used_data":20440,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94314652,"statfs":{"total":96636764160,"available":96578203648,"internally_reserved":0,"allocated":20930560,"data_stored":16483571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:28:11.584 INFO:tasks.ceph:pgid 1.f last_scrub_stamp 2024-03-30T13:27:55.467102+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=27, tm_sec=55, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=28, tm_sec=6, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T13:28:11.585 INFO:tasks.ceph:pgid 1.e last_scrub_stamp 2024-03-30T13:27:26.798199+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=27, tm_sec=26, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=28, tm_sec=6, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T13:28:11.585 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2024-03-30T13:28:03.597587+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=28, tm_sec=3, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=28, tm_sec=6, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T13:28:11.585 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-30T13:27:48.637751+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=27, tm_sec=48, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=28, tm_sec=6, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T13:28:11.585 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-03-30T13:28:06.927795+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=28, tm_sec=6, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=28, tm_sec=6, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T13:28:11.585 INFO:tasks.ceph:pgid 1.c last_scrub_stamp 2024-03-30T13:27:39.308113+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=27, tm_sec=39, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=28, tm_sec=6, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T13:28:11.585 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-03-30T13:27:12.596825+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=27, tm_sec=12, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=28, tm_sec=6, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T13:28:11.585 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-03-30T13:27:22.661428+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=27, tm_sec=22, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=28, tm_sec=6, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T13:28:11.585 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-03-30T13:27:41.018992+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=27, tm_sec=41, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=28, tm_sec=6, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T13:28:11.586 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-03-30T13:27:58.853242+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=27, tm_sec=58, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=28, tm_sec=6, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T13:28:11.586 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-03-30T13:27:17.528414+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=27, tm_sec=17, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=28, tm_sec=6, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T13:28:11.586 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-03-30T13:27:28.611732+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=27, tm_sec=28, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=28, tm_sec=6, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T13:28:11.586 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2024-03-30T13:27:42.923778+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=27, tm_sec=42, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=28, tm_sec=6, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T13:28:11.586 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2024-03-30T13:27:51.677029+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=27, tm_sec=51, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=28, tm_sec=6, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T13:28:11.586 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2024-03-30T13:27:35.881272+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=27, tm_sec=35, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=28, tm_sec=6, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T13:28:11.586 INFO:tasks.ceph:pgid 1.b last_scrub_stamp 2024-03-30T13:27:30.772331+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=27, tm_sec=30, tm_wday=5, tm_yday=90, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=30, tm_hour=13, tm_min=28, tm_sec=6, tm_wday=5, tm_yday=90, tm_isdst=0) 2024-03-30T13:28:11.586 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-30T13:28:31.587 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-30T13:28:31.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:31.714+0000 7f1ba9f57640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:31.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:31.730+0000 7f1ba9f57640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:31.901 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-03-30T13:28:31.902 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-03-30T13:28:31.918 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":1897,"stamp":"2024-03-30T13:28:30.356895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":276723559,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":468084,"kb_used_data":158292,"kb_used_omap":77,"kb_used_meta":309618,"kb_avail":754506636,"statfs":{"total":773094113280,"available":772614795264,"internally_reserved":0,"allocated":162091008,"data_stored":126352880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":79062,"internal_metadata":317049642},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002780"},"pg_stats":[{"pgid":"1.f","version":"22'6","reported_seq":2054,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-03-30T13:28:24.625669+0000","last_change":"2024-03-30T13:28:12.394436+0000","last_active":"2024-03-30T13:28:24.625669+0000","last_peered":"2024-03-30T13:28:24.625669+0000","last_clean":"2024-03-30T13:28:24.625669+0000","last_became_active":"2024-03-30T13:25:20.549084+0000","last_became_peered":"2024-03-30T13:25:20.549084+0000","last_unstale":"2024-03-30T13:28:24.625669+0000","last_undegraded":"2024-03-30T13:28:24.625669+0000","last_fullsized":"2024-03-30T13:28:24.625669+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:28:12.394161+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:28:12.394161+0000","last_clean_scrub_stamp":"2024-03-30T13:28:12.394161+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:38.153884+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":1781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":13764735,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'3","reported_seq":1942,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-03-30T13:28:24.626807+0000","last_change":"2024-03-30T13:28:10.968088+0000","last_active":"2024-03-30T13:28:24.626807+0000","last_peered":"2024-03-30T13:28:24.626807+0000","last_clean":"2024-03-30T13:28:24.626807+0000","last_became_active":"2024-03-30T13:26:16.583003+0000","last_became_peered":"2024-03-30T13:26:16.583003+0000","last_unstale":"2024-03-30T13:28:24.626807+0000","last_undegraded":"2024-03-30T13:28:24.626807+0000","last_fullsized":"2024-03-30T13:28:24.626807+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:28:10.967791+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:28:10.967791+0000","last_clean_scrub_stamp":"2024-03-30T13:28:10.967791+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:20.452011+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2655473,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":7966419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'9","reported_seq":212,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-03-30T13:28:24.626595+0000","last_change":"2024-03-30T13:28:14.621936+0000","last_active":"2024-03-30T13:28:24.626595+0000","last_peered":"2024-03-30T13:28:24.626595+0000","last_clean":"2024-03-30T13:28:24.626595+0000","last_became_active":"2024-03-30T13:27:07.482993+0000","last_became_peered":"2024-03-30T13:27:07.482993+0000","last_unstale":"2024-03-30T13:28:24.626595+0000","last_undegraded":"2024-03-30T13:28:24.626595+0000","last_fullsized":"2024-03-30T13:28:24.626595+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:28:14.621665+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:28:14.621665+0000","last_clean_scrub_stamp":"2024-03-30T13:28:14.621665+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:30.424073+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7815156,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":31260624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'6","reported_seq":191,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-03-30T13:28:24.626923+0000","last_change":"2024-03-30T13:28:12.572824+0000","last_active":"2024-03-30T13:28:24.626923+0000","last_peered":"2024-03-30T13:28:24.626923+0000","last_clean":"2024-03-30T13:28:24.626923+0000","last_became_active":"2024-03-30T13:26:16.577250+0000","last_became_peered":"2024-03-30T13:26:16.577250+0000","last_unstale":"2024-03-30T13:28:24.626923+0000","last_undegraded":"2024-03-30T13:28:24.626923+0000","last_fullsized":"2024-03-30T13:28:24.626923+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:28:12.572419+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:28:12.572419+0000","last_clean_scrub_stamp":"2024-03-30T13:28:12.572419+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:26.828996+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4588245,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":6756315,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"22'3","reported_seq":864,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-03-30T13:28:23.437585+0000","last_change":"2024-03-30T13:28:12.053558+0000","last_active":"2024-03-30T13:28:23.437585+0000","last_peered":"2024-03-30T13:28:23.437585+0000","last_clean":"2024-03-30T13:28:23.437585+0000","last_became_active":"2024-03-30T13:25:20.556165+0000","last_became_peered":"2024-03-30T13:25:20.556165+0000","last_unstale":"2024-03-30T13:28:23.437585+0000","last_undegraded":"2024-03-30T13:28:23.437585+0000","last_fullsized":"2024-03-30T13:28:23.437585+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:28:12.053314+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:28:12.053314+0000","last_clean_scrub_stamp":"2024-03-30T13:28:12.053314+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:26.292900+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"22'15","reported_seq":1993,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-03-30T13:28:24.627129+0000","last_change":"2024-03-30T13:28:09.486485+0000","last_active":"2024-03-30T13:28:24.627129+0000","last_peered":"2024-03-30T13:28:24.627129+0000","last_clean":"2024-03-30T13:28:24.627129+0000","last_became_active":"2024-03-30T13:25:20.551941+0000","last_became_peered":"2024-03-30T13:25:20.551941+0000","last_unstale":"2024-03-30T13:28:24.627129+0000","last_undegraded":"2024-03-30T13:28:24.627129+0000","last_fullsized":"2024-03-30T13:28:24.627129+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:28:09.486355+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:28:09.486355+0000","last_clean_scrub_stamp":"2024-03-30T13:28:09.486355+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:20.099070+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11394981,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":5340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11394981,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"22'18","reported_seq":198,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-03-30T13:28:24.627190+0000","last_change":"2024-03-30T13:28:13.589844+0000","last_active":"2024-03-30T13:28:24.627190+0000","last_peered":"2024-03-30T13:28:24.627190+0000","last_clean":"2024-03-30T13:28:24.627190+0000","last_became_active":"2024-03-30T13:26:16.583861+0000","last_became_peered":"2024-03-30T13:26:16.583861+0000","last_unstale":"2024-03-30T13:28:24.627190+0000","last_undegraded":"2024-03-30T13:28:24.627190+0000","last_fullsized":"2024-03-30T13:28:24.627190+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'18","last_scrub_stamp":"2024-03-30T13:28:13.589545+0000","last_deep_scrub":"22'18","last_deep_scrub_stamp":"2024-03-30T13:28:13.589545+0000","last_clean_scrub_stamp":"2024-03-30T13:28:13.589545+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:42.812401+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14638699,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":24,"num_write_kb":6991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":14638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'9","reported_seq":1301,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-03-30T13:28:23.831512+0000","last_change":"2024-03-30T13:28:09.811685+0000","last_active":"2024-03-30T13:28:23.831512+0000","last_peered":"2024-03-30T13:28:23.831512+0000","last_clean":"2024-03-30T13:28:23.831512+0000","last_became_active":"2024-03-30T13:27:07.598881+0000","last_became_peered":"2024-03-30T13:27:07.598881+0000","last_unstale":"2024-03-30T13:28:23.831512+0000","last_undegraded":"2024-03-30T13:28:23.831512+0000","last_fullsized":"2024-03-30T13:28:23.831512+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'9","last_scrub_stamp":"2024-03-30T13:28:09.811453+0000","last_deep_scrub":"22'9","last_deep_scrub_stamp":"2024-03-30T13:28:09.811453+0000","last_clean_scrub_stamp":"2024-03-30T13:28:09.811453+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:31.310342+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":12,"num_write_kb":3268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":58756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'6","reported_seq":2114,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-03-30T13:28:24.627468+0000","last_change":"2024-03-30T13:28:10.441039+0000","last_active":"2024-03-30T13:28:24.627468+0000","last_peered":"2024-03-30T13:28:24.627468+0000","last_clean":"2024-03-30T13:28:24.627468+0000","last_became_active":"2024-03-30T13:26:16.582448+0000","last_became_peered":"2024-03-30T13:26:16.582448+0000","last_unstale":"2024-03-30T13:28:24.627468+0000","last_undegraded":"2024-03-30T13:28:24.627468+0000","last_fullsized":"2024-03-30T13:28:24.627468+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:28:10.440921+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:28:10.440921+0000","last_clean_scrub_stamp":"2024-03-30T13:28:10.440921+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:20.236074+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5075648,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":76134720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'3","reported_seq":1752,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-03-30T13:28:24.625818+0000","last_change":"2024-03-30T13:28:09.920381+0000","last_active":"2024-03-30T13:28:24.625818+0000","last_peered":"2024-03-30T13:28:24.625818+0000","last_clean":"2024-03-30T13:28:24.625818+0000","last_became_active":"2024-03-30T13:24:29.442802+0000","last_became_peered":"2024-03-30T13:24:29.442802+0000","last_unstale":"2024-03-30T13:28:24.625818+0000","last_undegraded":"2024-03-30T13:28:24.625818+0000","last_fullsized":"2024-03-30T13:28:24.625818+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:28:09.920130+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:28:09.920130+0000","last_clean_scrub_stamp":"2024-03-30T13:28:09.920130+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:30.432988+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2336140,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":835,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2336140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"22'15","reported_seq":2062,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-03-30T13:28:24.626061+0000","last_change":"2024-03-30T13:28:11.421975+0000","last_active":"2024-03-30T13:28:24.626061+0000","last_peered":"2024-03-30T13:28:24.626061+0000","last_clean":"2024-03-30T13:28:24.626061+0000","last_became_active":"2024-03-30T13:24:29.447202+0000","last_became_peered":"2024-03-30T13:24:29.447202+0000","last_unstale":"2024-03-30T13:28:24.626061+0000","last_undegraded":"2024-03-30T13:28:24.626061+0000","last_fullsized":"2024-03-30T13:28:24.626061+0000","mapping_epoch":794,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":795,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'15","last_scrub_stamp":"2024-03-30T13:28:11.421691+0000","last_deep_scrub":"22'15","last_deep_scrub_stamp":"2024-03-30T13:28:11.421691+0000","last_clean_scrub_stamp":"2024-03-30T13:28:11.421691+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:19.462620+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12739541,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":20,"num_write_kb":6289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":12739541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,4],"acting":[6,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"22'24","reported_seq":2156,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-03-30T13:28:24.626701+0000","last_change":"2024-03-30T13:28:13.392738+0000","last_active":"2024-03-30T13:28:24.626701+0000","last_peered":"2024-03-30T13:28:24.626701+0000","last_clean":"2024-03-30T13:28:24.626701+0000","last_became_active":"2024-03-30T13:25:20.549465+0000","last_became_peered":"2024-03-30T13:25:20.549465+0000","last_unstale":"2024-03-30T13:28:24.626701+0000","last_undegraded":"2024-03-30T13:28:24.626701+0000","last_fullsized":"2024-03-30T13:28:24.626701+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'24","last_scrub_stamp":"2024-03-30T13:28:13.392396+0000","last_deep_scrub":"22'24","last_deep_scrub_stamp":"2024-03-30T13:28:13.392396+0000","last_clean_scrub_stamp":"2024-03-30T13:28:13.392396+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:38.869446+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19243751,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":32,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":38487502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'12","reported_seq":909,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-03-30T13:28:23.437587+0000","last_change":"2024-03-30T13:28:11.012383+0000","last_active":"2024-03-30T13:28:23.437587+0000","last_peered":"2024-03-30T13:28:23.437587+0000","last_clean":"2024-03-30T13:28:23.437587+0000","last_became_active":"2024-03-30T13:25:20.549427+0000","last_became_peered":"2024-03-30T13:25:20.549427+0000","last_unstale":"2024-03-30T13:28:23.437587+0000","last_undegraded":"2024-03-30T13:28:23.437587+0000","last_fullsized":"2024-03-30T13:28:23.437587+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:28:11.012107+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:28:11.012107+0000","last_clean_scrub_stamp":"2024-03-30T13:28:11.012107+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:13.736441+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9159683,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":16,"num_write_kb":3914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'3","reported_seq":189,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-03-30T13:28:24.626627+0000","last_change":"2024-03-30T13:28:11.536748+0000","last_active":"2024-03-30T13:28:24.626627+0000","last_peered":"2024-03-30T13:28:24.626627+0000","last_clean":"2024-03-30T13:28:24.626627+0000","last_became_active":"2024-03-30T13:26:16.578484+0000","last_became_peered":"2024-03-30T13:26:16.578484+0000","last_unstale":"2024-03-30T13:28:24.626627+0000","last_undegraded":"2024-03-30T13:28:24.626627+0000","last_fullsized":"2024-03-30T13:28:24.626627+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'3","last_scrub_stamp":"2024-03-30T13:28:11.536490+0000","last_deep_scrub":"22'3","last_deep_scrub_stamp":"2024-03-30T13:28:11.536490+0000","last_clean_scrub_stamp":"2024-03-30T13:28:11.536490+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:13.543254+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2487403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2487403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'12","reported_seq":924,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-03-30T13:28:23.437521+0000","last_change":"2024-03-30T13:28:13.036790+0000","last_active":"2024-03-30T13:28:23.437521+0000","last_peered":"2024-03-30T13:28:23.437521+0000","last_clean":"2024-03-30T13:28:23.437521+0000","last_became_active":"2024-03-30T13:25:20.556235+0000","last_became_peered":"2024-03-30T13:25:20.556235+0000","last_unstale":"2024-03-30T13:28:23.437521+0000","last_undegraded":"2024-03-30T13:28:23.437521+0000","last_fullsized":"2024-03-30T13:28:23.437521+0000","mapping_epoch":816,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":817,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'12","last_scrub_stamp":"2024-03-30T13:28:13.036505+0000","last_deep_scrub":"22'12","last_deep_scrub_stamp":"2024-03-30T13:28:13.036505+0000","last_clean_scrub_stamp":"2024-03-30T13:28:13.036505+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:17.411837+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9075648,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":16,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'6","reported_seq":1164,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-03-30T13:28:23.831535+0000","last_change":"2024-03-30T13:28:08.779998+0000","last_active":"2024-03-30T13:28:23.831535+0000","last_peered":"2024-03-30T13:28:23.831535+0000","last_clean":"2024-03-30T13:28:23.831535+0000","last_became_active":"2024-03-30T13:27:07.478858+0000","last_became_peered":"2024-03-30T13:27:07.478858+0000","last_unstale":"2024-03-30T13:28:23.831535+0000","last_undegraded":"2024-03-30T13:28:23.831535+0000","last_fullsized":"2024-03-30T13:28:23.831535+0000","mapping_epoch":861,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":862,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'6","last_scrub_stamp":"2024-03-30T13:28:08.779823+0000","last_deep_scrub":"22'6","last_deep_scrub_stamp":"2024-03-30T13:28:08.779823+0000","last_clean_scrub_stamp":"2024-03-30T13:28:08.779823+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T13:29:18.113739+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5226911,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":16,"stat_sum":{"num_bytes":120605382,"num_objects":50,"num_object_clones":0,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":84,"num_read_kb":34,"num_write":200,"num_write_kb":54950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":112,"num_bytes_recovered":276723559,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":96751616,"data_stored":96751616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":48,"acting":48,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":722,"seq":3100966387793,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60280,"kb_used_data":23084,"kb_used_omap":9,"kb_used_meta":37174,"kb_avail":94311560,"statfs":{"total":96636764160,"available":96575037440,"internally_reserved":0,"allocated":23638016,"data_stored":19170750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9342,"internal_metadata":38067074},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":816,"seq":3504693313577,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69632,"kb_used_data":27828,"kb_used_omap":12,"kb_used_meta":41779,"kb_avail":94302208,"statfs":{"total":96636764160,"available":96565460992,"internally_reserved":0,"allocated":28495872,"data_stored":24028606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12654,"internal_metadata":42782354},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":794,"seq":3410204033075,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60056,"kb_used_data":19532,"kb_used_omap":15,"kb_used_meta":40496,"kb_avail":94311784,"statfs":{"total":96636764160,"available":96575266816,"internally_reserved":0,"allocated":20000768,"data_stored":15533502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15941,"internal_metadata":41468347},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509840,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53800,"kb_used_data":14300,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94318040,"statfs":{"total":96636764160,"available":96581672960,"internally_reserved":0,"allocated":14643200,"data_stored":10175934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":861,"seq":3697966841876,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51220,"kb_used_data":13576,"kb_used_omap":12,"kb_used_meta":37619,"kb_avail":94320620,"statfs":{"total":96636764160,"available":96584314880,"internally_reserved":0,"allocated":13901824,"data_stored":9434558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12649,"internal_metadata":38522519},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509841,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56816,"kb_used_data":21476,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94315024,"statfs":{"total":96636764160,"available":96578584576,"internally_reserved":0,"allocated":21991424,"data_stored":17524158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":36174273},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":837,"seq":3594887626782,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59056,"kb_used_data":18020,"kb_used_omap":17,"kb_used_meta":41006,"kb_avail":94312784,"statfs":{"total":96636764160,"available":96576290816,"internally_reserved":0,"allocated":18452480,"data_stored":13985214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17588,"internal_metadata":41990988},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":747,"seq":3208340570183,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57224,"kb_used_data":20476,"kb_used_omap":7,"kb_used_meta":36728,"kb_avail":94314616,"statfs":{"total":96636764160,"available":96578166784,"internally_reserved":0,"allocated":20967424,"data_stored":16500158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7694,"internal_metadata":37609970},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":13824000,"data_stored":13824000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5734400,"data_stored":5734400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6475776,"data_stored":6475776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12800000,"data_stored":12800000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11833344,"data_stored":11833344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20328448,"data_stored":20328448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15470592,"data_stored":15470592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":10285056,"data_stored":10285056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-30T13:28:31.920 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-30T13:28:32.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:32.030+0000 7f99ef8ba640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:32.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-03-30T13:28:32.050+0000 7f99ef8ba640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-30T13:28:32.255 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa/tasks/rados.py", line 299, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa/tasks/rados.py", line 280, in thread run.wait(tests.values()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' 2024-03-30T13:28:32.256 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-30T13:28:32.257 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-30T13:28:32.257 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-30T13:28:32.257 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:28:32.351 INFO:tasks.ceph.osd.0:Stopped 2024-03-30T13:28:32.352 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-30T13:28:32.352 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:28:32.443 INFO:tasks.ceph.osd.1:Stopped 2024-03-30T13:28:32.443 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-30T13:28:32.443 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:28:34.244 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-03-30T13:28:38.545 INFO:tasks.ceph.osd.2:Stopped 2024-03-30T13:28:38.545 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-30T13:28:38.545 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:28:38.603 INFO:tasks.ceph.osd.3:Stopped 2024-03-30T13:28:38.603 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-30T13:28:38.603 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:28:38.688 INFO:tasks.ceph.osd.4:Stopped 2024-03-30T13:28:38.688 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-30T13:28:38.688 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:28:38.752 INFO:tasks.ceph.osd.5:Stopped 2024-03-30T13:28:38.752 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-30T13:28:38.752 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:28:38.824 INFO:tasks.ceph.osd.6:Stopped 2024-03-30T13:28:38.824 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-30T13:28:38.824 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:28:38.925 INFO:tasks.ceph.osd.7:Stopped 2024-03-30T13:28:38.925 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-30T13:28:38.925 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-30T13:28:38.925 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:28:38.954 INFO:tasks.ceph.mgr.y:Stopped 2024-03-30T13:28:38.955 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-30T13:28:38.955 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:28:39.008 INFO:tasks.ceph.mgr.x:Stopped 2024-03-30T13:28:39.008 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-30T13:28:39.008 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-30T13:28:39.008 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:28:39.027 INFO:tasks.ceph.mon.a:Stopped 2024-03-30T13:28:39.027 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-30T13:28:39.027 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:28:39.085 INFO:tasks.ceph.mon.c:Stopped 2024-03-30T13:28:39.085 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-30T13:28:39.085 INFO:teuthology.orchestra.run:waiting for 300 2024-03-30T13:28:39.103 INFO:tasks.ceph.mon.b:Stopped 2024-03-30T13:28:39.103 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-30T13:28:39.104 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | head -n 1 2024-03-30T13:28:39.144 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi067.front.sepia.ceph.com 2024-03-30T13:28:39.145 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-30T13:28:39.245 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-03-30T13:28:39.448 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi067.front.sepia.ceph.com 2024-03-30T13:28:39.449 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-30T13:28:39.647 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi067.front.sepia.ceph.com 2024-03-30T13:28:39.647 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-30T13:28:39.832 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi067.front.sepia.ceph.com 2024-03-30T13:28:39.836 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-30T13:28:40.108 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi082.front.sepia.ceph.com 2024-03-30T13:28:40.108 DEBUG:teuthology.orchestra.run.smithi082:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-30T13:28:40.420 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi082.front.sepia.ceph.com 2024-03-30T13:28:40.420 DEBUG:teuthology.orchestra.run.smithi082:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-30T13:28:40.627 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi082.front.sepia.ceph.com 2024-03-30T13:28:40.628 DEBUG:teuthology.orchestra.run.smithi082:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-30T13:28:40.810 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi082.front.sepia.ceph.com 2024-03-30T13:28:40.810 DEBUG:teuthology.orchestra.run.smithi082:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-30T13:28:40.979 INFO:tasks.ceph:Archiving mon data... 2024-03-30T13:28:40.981 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630730/data/mon.a.tgz 2024-03-30T13:28:40.981 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-03-30T13:28:40.987 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.yptrVdDD9V 2024-03-30T13:28:40.988 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.yptrVdDD9V 2024-03-30T13:28:41.178 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.yptrVdDD9V 2024-03-30T13:28:41.241 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.yptrVdDD9V is 617KB 2024-03-30T13:28:41.306 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.yptrVdDD9V 2024-03-30T13:28:41.311 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630730/data/mon.c.tgz 2024-03-30T13:28:41.312 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2024-03-30T13:28:41.360 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.rXQTQg5q1y 2024-03-30T13:28:41.360 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.rXQTQg5q1y 2024-03-30T13:28:41.554 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.rXQTQg5q1y 2024-03-30T13:28:41.621 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.rXQTQg5q1y is 640KB 2024-03-30T13:28:41.686 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.rXQTQg5q1y 2024-03-30T13:28:41.691 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630730/data/mon.b.tgz 2024-03-30T13:28:41.691 DEBUG:teuthology.orchestra.run.smithi082:> mktemp 2024-03-30T13:28:41.697 INFO:teuthology.orchestra.run.smithi082.stdout:/tmp/tmp.JYVCX8pW6s 2024-03-30T13:28:41.697 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.JYVCX8pW6s 2024-03-30T13:28:41.888 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 0666 /tmp/tmp.JYVCX8pW6s 2024-03-30T13:28:41.954 DEBUG:teuthology.orchestra.remote:smithi082:/tmp/tmp.JYVCX8pW6s is 640KB 2024-03-30T13:28:42.015 DEBUG:teuthology.orchestra.run.smithi082:> rm -fr /tmp/tmp.JYVCX8pW6s 2024-03-30T13:28:42.021 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-30T13:28:42.021 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-30T13:28:42.023 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-30T13:28:42.084 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-30T13:28:42.126 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-30T13:28:42.126 INFO:tasks.ceph:Archiving crash dumps... 2024-03-30T13:28:42.128 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630730/remote/smithi067/crash 2024-03-30T13:28:42.129 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-30T13:28:42.142 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630730/remote/smithi082/crash 2024-03-30T13:28:42.143 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-30T13:28:42.182 INFO:tasks.ceph:Compressing logs... 2024-03-30T13:28:42.182 DEBUG:teuthology.orchestra.run.smithi067:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-30T13:28:42.189 DEBUG:teuthology.orchestra.run.smithi082:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-30T13:28:42.213 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35457.log 2024-03-30T13:28:42.213 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62197.log 2024-03-30T13:28:42.214 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52700.log 2024-03-30T13:28:42.214 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35457.log: gzip/var/log/ceph/ceph-client.admin.62197.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.62353.log 2024-03-30T13:28:42.214 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% 46.7% -- replaced with /var/log/ceph/ceph-client.admin.62197.log.gz -- replaced with /var/log/ceph/ceph-client.admin.35457.log.gz 2024-03-30T13:28:42.214 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.215 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56740.log 2024-03-30T13:28:42.215 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.52700.log.gz 2024-03-30T13:28:42.215 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54786.log 2024-03-30T13:28:42.215 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62353.log: /var/log/ceph/ceph-client.admin.56740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46530.log 2024-03-30T13:28:42.216 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.56740.log.gz 2024-03-30T13:28:42.216 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.62353.log.gz 2024-03-30T13:28:42.217 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50319.log 2024-03-30T13:28:42.217 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54786.log: 46.7%/var/log/ceph/ceph-client.admin.46530.log: -- replaced with /var/log/ceph/ceph-client.admin.54786.log.gz 2024-03-30T13:28:42.217 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37333.log 2024-03-30T13:28:42.217 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.46530.log.gz 2024-03-30T13:28:42.217 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36576.log 2024-03-30T13:28:42.217 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.50319.log.gz 2024-03-30T13:28:42.218 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49431.log 2024-03-30T13:28:42.218 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.37333.log.gz 2024-03-30T13:28:42.218 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56636.log 2024-03-30T13:28:42.218 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.36576.log.gz 2024-03-30T13:28:42.221 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49717.log 2024-03-30T13:28:42.221 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49431.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49431.log.gz 2024-03-30T13:28:42.221 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30607.log 2024-03-30T13:28:42.221 INFO:teuthology.orchestra.run.smithi067.stderr: 48.1% -- replaced with /var/log/ceph/ceph-client.admin.56636.log.gz 2024-03-30T13:28:42.221 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66160.log 2024-03-30T13:28:42.221 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49717.log.gz 2024-03-30T13:28:42.221 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30607.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.30607.log.gz 2024-03-30T13:28:42.221 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48443.log 2024-03-30T13:28:42.222 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65169.log 2024-03-30T13:28:42.222 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.66160.log.gz 2024-03-30T13:28:42.222 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61729.log 2024-03-30T13:28:42.222 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.48443.log.gz 2024-03-30T13:28:42.222 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58462.log 2024-03-30T13:28:42.222 INFO:teuthology.orchestra.run.smithi067.stderr: 44.9% -- replaced with /var/log/ceph/ceph-client.admin.65169.log.gz 2024-03-30T13:28:42.222 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35015.log 2024-03-30T13:28:42.222 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.61729.log.gz 2024-03-30T13:28:42.222 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32671.log 2024-03-30T13:28:42.222 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.58462.log.gz 2024-03-30T13:28:42.228 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.35015.log: /var/log/ceph/ceph-client.admin.64047.log 2024-03-30T13:28:42.228 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.35015.log.gz 2024-03-30T13:28:42.228 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33061.log 2024-03-30T13:28:42.228 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.32671.log.gz 2024-03-30T13:28:42.228 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64047.log: 46.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.64047.log.gz 2024-03-30T13:28:42.228 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.33529.log 2024-03-30T13:28:42.228 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64834.log 2024-03-30T13:28:42.228 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33061.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.33061.log.gz 2024-03-30T13:28:42.228 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33529.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.33529.log.gz 2024-03-30T13:28:42.228 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52151.log 2024-03-30T13:28:42.229 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49509.log 2024-03-30T13:28:42.229 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64834.log.gz 2024-03-30T13:28:42.229 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55254.log 2024-03-30T13:28:42.229 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52151.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.52151.log.gz 2024-03-30T13:28:42.229 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64307.log 2024-03-30T13:28:42.229 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49509.log.gz 2024-03-30T13:28:42.229 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64678.log 2024-03-30T13:28:42.229 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.55254.log.gz 2024-03-30T13:28:42.229 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41084.log 2024-03-30T13:28:42.229 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64307.log.gz 2024-03-30T13:28:42.229 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54343.log 2024-03-30T13:28:42.229 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64678.log.gz 2024-03-30T13:28:42.229 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33243.log 2024-03-30T13:28:42.229 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.41084.log.gz 2024-03-30T13:28:42.229 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37515.log 2024-03-30T13:28:42.230 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54343.log.gz 2024-03-30T13:28:42.230 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33243.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-30T13:28:42.230 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.33243.log.gz 2024-03-30T13:28:42.230 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53458.log 2024-03-30T13:28:42.230 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.37515.log.gz 2024-03-30T13:28:42.230 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54968.log 2024-03-30T13:28:42.230 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53796.log 2024-03-30T13:28:42.230 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53458.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.53458.log.gz 2024-03-30T13:28:42.230 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39184.log 2024-03-30T13:28:42.230 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54968.log.gz 2024-03-30T13:28:42.243 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58618.log 2024-03-30T13:28:42.243 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53796.log: /var/log/ceph/ceph-client.admin.39184.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.53796.log.gz 2024-03-30T13:28:42.243 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.39184.log.gz 2024-03-30T13:28:42.243 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49825.log 2024-03-30T13:28:42.243 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33009.log 2024-03-30T13:28:42.244 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.58618.log.gz 2024-03-30T13:28:42.244 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53822.log 2024-03-30T13:28:42.244 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49825.log.gz 2024-03-30T13:28:42.244 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54655.log 2024-03-30T13:28:42.244 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33009.log: /var/log/ceph/ceph-client.admin.53822.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.33009.log.gz 2024-03-30T13:28:42.244 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50215.log 2024-03-30T13:28:42.244 INFO:teuthology.orchestra.run.smithi067.stderr: 44.7% -- replaced with /var/log/ceph/ceph-client.admin.53822.log.gz 2024-03-30T13:28:42.244 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54655.log: 46.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.54655.log.gz 2024-03-30T13:28:42.244 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53197.log 2024-03-30T13:28:42.244 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46192.log 2024-03-30T13:28:42.244 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50215.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.50215.log.gz 2024-03-30T13:28:42.244 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64756.log 2024-03-30T13:28:42.244 INFO:teuthology.orchestra.run.smithi067.stderr: 47.7% -- replaced with /var/log/ceph/ceph-client.admin.53197.log.gz 2024-03-30T13:28:42.244 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53978.log 2024-03-30T13:28:42.244 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.46192.log.gz 2024-03-30T13:28:42.245 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56272.log 2024-03-30T13:28:42.245 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64756.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64756.log.gz 2024-03-30T13:28:42.245 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53978.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.53978.log.gz 2024-03-30T13:28:42.245 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48807.log 2024-03-30T13:28:42.245 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60528.log 2024-03-30T13:28:42.245 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56272.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.56272.log.gz 2024-03-30T13:28:42.245 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40694.log 2024-03-30T13:28:42.245 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.48807.log.gz 2024-03-30T13:28:42.245 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40330.log 2024-03-30T13:28:42.245 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60528.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.60528.log.gz 2024-03-30T13:28:42.245 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40694.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.40694.log.gz 2024-03-30T13:28:42.245 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32775.log 2024-03-30T13:28:42.245 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31848.log 2024-03-30T13:28:42.245 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40330.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.40330.log.gz 2024-03-30T13:28:42.245 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56610.log 2024-03-30T13:28:42.246 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.32775.log.gz 2024-03-30T13:28:42.246 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57468.log 2024-03-30T13:28:42.246 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31848.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.31848.log.gz 2024-03-30T13:28:42.246 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56610.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.56610.log.gz 2024-03-30T13:28:42.246 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33477.log 2024-03-30T13:28:42.246 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36628.log 2024-03-30T13:28:42.246 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57468.log: 47.7% -- replaced with /var/log/ceph/ceph-client.admin.57468.log.gz 2024-03-30T13:28:42.246 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36732.log 2024-03-30T13:28:42.246 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.33477.log.gz 2024-03-30T13:28:42.246 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30793.log 2024-03-30T13:28:42.246 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36628.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.36628.log.gz 2024-03-30T13:28:42.246 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36732.log: 46.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.36732.log.gz 2024-03-30T13:28:42.246 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.30799.log 2024-03-30T13:28:42.246 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62015.log 2024-03-30T13:28:42.246 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.30793.log.gz 2024-03-30T13:28:42.247 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40928.log 2024-03-30T13:28:42.247 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30799.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.30799.log.gz 2024-03-30T13:28:42.247 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62015.log: gzip 46.7% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.62015.log.gz -- 2024-03-30T13:28:42.247 INFO:teuthology.orchestra.run.smithi067.stderr: /var/log/ceph/ceph-client.admin.32697.log 2024-03-30T13:28:42.247 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38687.log 2024-03-30T13:28:42.247 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.40928.log.gz 2024-03-30T13:28:42.247 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41736.log 2024-03-30T13:28:42.247 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.32697.log.gz 2024-03-30T13:28:42.247 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.38687.log: /var/log/ceph/ceph-client.admin.49799.log 2024-03-30T13:28:42.247 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.38687.log.gz 2024-03-30T13:28:42.247 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41736.log: gzip -5 46.7% --verbose -- replaced with /var/log/ceph/ceph-client.admin.41736.log.gz -- 2024-03-30T13:28:42.247 INFO:teuthology.orchestra.run.smithi067.stderr: /var/log/ceph/ceph-client.admin.35770.log 2024-03-30T13:28:42.247 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64730.log 2024-03-30T13:28:42.247 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49799.log.gz 2024-03-30T13:28:42.248 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58514.log 2024-03-30T13:28:42.248 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35770.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.35770.log.gz 2024-03-30T13:28:42.248 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64730.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.64730.log.gz 2024-03-30T13:28:42.248 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40044.log 2024-03-30T13:28:42.248 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30558.log 2024-03-30T13:28:42.248 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.58514.log.gz 2024-03-30T13:28:42.248 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46738.log 2024-03-30T13:28:42.248 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.40044.log.gz 2024-03-30T13:28:42.248 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39418.log 2024-03-30T13:28:42.248 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30558.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.30558.log.gz 2024-03-30T13:28:42.248 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46738.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.46738.log.gzgzip 2024-03-30T13:28:42.248 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48989.log 2024-03-30T13:28:42.248 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61833.log 2024-03-30T13:28:42.248 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.39418.log.gz 2024-03-30T13:28:42.248 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43847.log 2024-03-30T13:28:42.248 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48989.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.48989.log.gz 2024-03-30T13:28:42.249 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61833.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.61833.log.gz 2024-03-30T13:28:42.249 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50475.log 2024-03-30T13:28:42.264 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25220.log 2024-03-30T13:28:42.264 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25148.log 2024-03-30T13:28:42.264 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.25220.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.25220.log.gz 2024-03-30T13:28:42.264 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-30T13:28:42.264 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.25148.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-30T13:28:42.264 INFO:teuthology.orchestra.run.smithi082.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.25148.log.gz 2024-03-30T13:28:42.264 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25076.log 2024-03-30T13:28:42.264 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-osd.7.log: /var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25124.log 2024-03-30T13:28:42.265 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.25076.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-30T13:28:42.265 INFO:teuthology.orchestra.run.smithi082.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.25076.log.gz 2024-03-30T13:28:42.265 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.25124.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.25124.log.gz 2024-03-30T13:28:42.265 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25100.log 2024-03-30T13:28:42.265 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-30T13:28:42.265 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.25100.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.25100.log.gz 2024-03-30T13:28:42.265 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-30T13:28:42.265 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25196.log 2024-03-30T13:28:42.265 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-30T13:28:42.265 INFO:teuthology.orchestra.run.smithi082.stderr:gzip/var/log/ceph/ceph-client.admin.25196.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.25244.log 47.2% -- replaced with /var/log/ceph/ceph-client.admin.25196.log.gz 2024-03-30T13:28:42.265 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-30T13:28:42.265 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-30T13:28:42.265 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.25244.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-30T13:28:42.266 INFO:teuthology.orchestra.run.smithi082.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.25244.log.gz 2024-03-30T13:28:42.266 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25172.log 2024-03-30T13:28:42.275 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-osd.4.log: 90.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-30T13:28:42.275 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/ceph-client.admin.25172.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.25172.log.gz 2024-03-30T13:28:42.275 INFO:teuthology.orchestra.run.smithi082.stderr: 86.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-30T13:28:42.371 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56584.log 2024-03-30T13:28:42.371 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43847.log.gz 2024-03-30T13:28:42.371 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34495.log 2024-03-30T13:28:42.372 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.50475.log.gz 2024-03-30T13:28:42.372 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60216.log 2024-03-30T13:28:42.372 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56584.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.56584.log.gz 2024-03-30T13:28:42.372 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34495.log: 46.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.34495.log.gz 2024-03-30T13:28:42.372 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.60372.log 2024-03-30T13:28:42.372 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49561.log 2024-03-30T13:28:42.372 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.60216.log.gz 2024-03-30T13:28:42.372 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52911.log 2024-03-30T13:28:42.372 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60372.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.60372.log.gz 2024-03-30T13:28:42.372 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49561.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49561.log.gz 2024-03-30T13:28:42.372 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48833.log 2024-03-30T13:28:42.372 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52911.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-30T13:28:42.372 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.52911.log.gz 2024-03-30T13:28:42.372 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64073.log 2024-03-30T13:28:42.372 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.48833.log.gz 2024-03-30T13:28:42.373 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64958.log 2024-03-30T13:28:42.373 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.64073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36108.log 2024-03-30T13:28:42.373 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64073.log.gz 2024-03-30T13:28:42.373 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64958.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.64958.log.gz 2024-03-30T13:28:42.373 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48469.log 2024-03-30T13:28:42.373 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58852.log 2024-03-30T13:28:42.373 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.36108.log.gz 2024-03-30T13:28:42.373 INFO:teuthology.orchestra.run.smithi067.stderr: 94.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-30T13:28:42.373 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48469.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.48469.log.gz 2024-03-30T13:28:42.373 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45880.log 2024-03-30T13:28:42.373 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47922.log 2024-03-30T13:28:42.373 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.58852.log.gz 2024-03-30T13:28:42.373 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45880.log: gzip 46.7% -- replaced with /var/log/ceph/ceph-client.admin.45880.log.gz 2024-03-30T13:28:42.373 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48000.log 2024-03-30T13:28:42.373 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62249.log 2024-03-30T13:28:42.374 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.47922.log.gz 2024-03-30T13:28:42.374 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48000.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.48000.log.gz 2024-03-30T13:28:42.374 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64177.log 2024-03-30T13:28:42.374 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46764.log 2024-03-30T13:28:42.374 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.62249.log.gz 2024-03-30T13:28:42.374 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64177.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64177.log.gz 2024-03-30T13:28:42.374 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41294.log 2024-03-30T13:28:42.374 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64950.log 2024-03-30T13:28:42.374 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.46764.log.gz 2024-03-30T13:28:42.374 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41294.log: gzip 46.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.41294.log.gz 2024-03-30T13:28:42.374 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.64522.log 2024-03-30T13:28:42.374 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46348.log 2024-03-30T13:28:42.374 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64950.log.gz 2024-03-30T13:28:42.374 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64522.log: gzip 46.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.64522.log.gz 2024-03-30T13:28:42.374 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48781.log 2024-03-30T13:28:42.375 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54812.log 2024-03-30T13:28:42.375 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.46348.log.gz 2024-03-30T13:28:42.375 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65926.log 2024-03-30T13:28:42.375 INFO:teuthology.orchestra.run.smithi067.stderr: 47.7% -- replaced with /var/log/ceph/ceph-client.admin.48781.log.gz 2024-03-30T13:28:42.375 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54812.log: gzip 46.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.54812.log.gz --verbose 2024-03-30T13:28:42.375 INFO:teuthology.orchestra.run.smithi067.stderr: -- /var/log/ceph/ceph-client.admin.46922.log 2024-03-30T13:28:42.375 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54629.log 2024-03-30T13:28:42.375 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.65926.log.gz 2024-03-30T13:28:42.375 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46922.log: 46.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.46922.log.gz 2024-03-30T13:28:42.375 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.63891.log 2024-03-30T13:28:42.375 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50657.log 2024-03-30T13:28:42.375 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54629.log.gz 2024-03-30T13:28:42.375 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64255.log 2024-03-30T13:28:42.375 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.63891.log.gz 2024-03-30T13:28:42.375 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32359.log 2024-03-30T13:28:42.376 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.50657.log.gz 2024-03-30T13:28:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39103.log 2024-03-30T13:28:42.376 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64255.log.gz 2024-03-30T13:28:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33347.log 2024-03-30T13:28:42.376 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.32359.log.gz 2024-03-30T13:28:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44029.log 2024-03-30T13:28:42.376 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.39103.log.gz 2024-03-30T13:28:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49301.log 2024-03-30T13:28:42.376 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.33347.log.gz 2024-03-30T13:28:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34885.log 2024-03-30T13:28:42.376 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.44029.log.gz 2024-03-30T13:28:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62171.log 2024-03-30T13:28:42.376 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49301.log.gz 2024-03-30T13:28:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57390.log 2024-03-30T13:28:42.376 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.34885.log.gz 2024-03-30T13:28:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36602.log 2024-03-30T13:28:42.377 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.62171.log.gz 2024-03-30T13:28:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39025.log 2024-03-30T13:28:42.377 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.57390.log.gz 2024-03-30T13:28:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55020.log 2024-03-30T13:28:42.377 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.36602.log.gz 2024-03-30T13:28:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61677.log 2024-03-30T13:28:42.377 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.39025.log.gz 2024-03-30T13:28:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40460.log 2024-03-30T13:28:42.377 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.55020.log.gz 2024-03-30T13:28:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31433.log 2024-03-30T13:28:42.377 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.61677.log.gz 2024-03-30T13:28:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34859.log 2024-03-30T13:28:42.377 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.40460.log.gz 2024-03-30T13:28:42.377 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60034.log 2024-03-30T13:28:42.377 INFO:teuthology.orchestra.run.smithi067.stderr: 45.4% -- replaced with /var/log/ceph/ceph-client.admin.31433.log.gz 2024-03-30T13:28:42.378 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48416.log 2024-03-30T13:28:42.378 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.34859.log.gz 2024-03-30T13:28:42.378 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49929.log 2024-03-30T13:28:42.378 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.60034.log.gz 2024-03-30T13:28:42.378 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61885.log 2024-03-30T13:28:42.378 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.48416.log.gz 2024-03-30T13:28:42.378 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50241.log 2024-03-30T13:28:42.378 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49929.log.gz 2024-03-30T13:28:42.378 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50163.log 2024-03-30T13:28:42.378 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.61885.log.gz 2024-03-30T13:28:42.378 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31312.log 2024-03-30T13:28:42.378 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.50241.log.gz 2024-03-30T13:28:42.378 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41424.log 2024-03-30T13:28:42.378 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.50163.log.gz 2024-03-30T13:28:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65666.log 2024-03-30T13:28:42.379 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.31312.log.gz 2024-03-30T13:28:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52099.log 2024-03-30T13:28:42.379 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.41424.log.gz 2024-03-30T13:28:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38427.log 2024-03-30T13:28:42.379 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.65666.log.gz 2024-03-30T13:28:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38765.log 2024-03-30T13:28:42.379 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.52099.log.gz 2024-03-30T13:28:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64548.log 2024-03-30T13:28:42.379 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.38427.log.gz 2024-03-30T13:28:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34911.log 2024-03-30T13:28:42.379 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.38765.log.gz 2024-03-30T13:28:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40902.log 2024-03-30T13:28:42.379 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64548.log.gz 2024-03-30T13:28:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55332.log 2024-03-30T13:28:42.379 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.34911.log.gz 2024-03-30T13:28:42.380 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46868.log 2024-03-30T13:28:42.380 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.40902.log.gz 2024-03-30T13:28:42.380 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39836.log 2024-03-30T13:28:42.380 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.55332.log.gz 2024-03-30T13:28:42.380 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34625.log 2024-03-30T13:28:42.380 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.46868.log.gz 2024-03-30T13:28:42.380 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64333.log 2024-03-30T13:28:42.380 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.39836.log.gz 2024-03-30T13:28:42.380 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62275.log 2024-03-30T13:28:42.380 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.34625.log.gz 2024-03-30T13:28:42.380 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52858.log 2024-03-30T13:28:42.380 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64333.log.gz 2024-03-30T13:28:42.380 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62067.log 2024-03-30T13:28:42.381 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.62275.log.gz 2024-03-30T13:28:42.381 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43066.log 2024-03-30T13:28:42.381 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.52858.log.gz 2024-03-30T13:28:42.381 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45463.log 2024-03-30T13:28:42.381 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.62067.log.gz 2024-03-30T13:28:42.381 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53900.log 46.7% 2024-03-30T13:28:42.381 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43066.log.gz 2024-03-30T13:28:42.381 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61391.log 2024-03-30T13:28:42.381 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.45463.log.gz 2024-03-30T13:28:42.381 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39730.log 2024-03-30T13:28:42.381 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.53900.log.gz 2024-03-30T13:28:42.381 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49903.log 2024-03-30T13:28:42.381 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.61391.log.gz 2024-03-30T13:28:42.382 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55072.log 2024-03-30T13:28:42.382 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.39730.log.gz 2024-03-30T13:28:42.382 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46270.log 2024-03-30T13:28:42.382 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.49903.log.gz 2024-03-30T13:28:42.382 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61287.log 2024-03-30T13:28:42.382 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.55072.log.gz 2024-03-30T13:28:42.382 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43248.log 2024-03-30T13:28:42.382 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.46270.log.gz 2024-03-30T13:28:42.382 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54057.log 2024-03-30T13:28:42.382 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.61287.log.gz 2024-03-30T13:28:42.382 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53093.log 2024-03-30T13:28:42.382 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.43248.log.gz 2024-03-30T13:28:42.382 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39966.log 2024-03-30T13:28:42.382 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54057.log.gz 2024-03-30T13:28:42.382 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42832.log 2024-03-30T13:28:42.383 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.53093.log.gz 2024-03-30T13:28:42.383 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34547.log 2024-03-30T13:28:42.383 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.39966.log.gz 2024-03-30T13:28:42.383 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32801.log 2024-03-30T13:28:42.383 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.42832.log.gz 2024-03-30T13:28:42.383 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60450.log 2024-03-30T13:28:42.383 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.34547.log.gz 2024-03-30T13:28:42.383 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53145.log 2024-03-30T13:28:42.383 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.32801.log.gz 2024-03-30T13:28:42.383 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42936.log 2024-03-30T13:28:42.383 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.60450.log.gz 2024-03-30T13:28:42.383 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31027.log 2024-03-30T13:28:42.383 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.53145.log.gz 2024-03-30T13:28:42.383 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57494.log 2024-03-30T13:28:42.383 INFO:teuthology.orchestra.run.smithi067.stderr: 45.5% -- replaced with /var/log/ceph/ceph-client.admin.42936.log.gz 2024-03-30T13:28:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40616.log 2024-03-30T13:28:42.384 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.31027.log.gz 2024-03-30T13:28:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38193.log 2024-03-30T13:28:42.384 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.57494.log.gz 2024-03-30T13:28:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37463.log 2024-03-30T13:28:42.384 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.40616.log.gz 2024-03-30T13:28:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60658.log 2024-03-30T13:28:42.384 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.38193.log.gz 2024-03-30T13:28:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64574.log 2024-03-30T13:28:42.384 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.37463.log.gz 2024-03-30T13:28:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50345.log 2024-03-30T13:28:42.384 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.60658.log.gz 2024-03-30T13:28:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53275.log 2024-03-30T13:28:42.384 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64574.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64574.log.gz 2024-03-30T13:28:42.385 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58956.log 2024-03-30T13:28:42.385 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.50345.log.gz 2024-03-30T13:28:42.385 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50683.log 2024-03-30T13:28:42.385 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.53275.log.gz 2024-03-30T13:28:42.385 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53848.log 2024-03-30T13:28:42.385 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.58956.log.gz 2024-03-30T13:28:42.385 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53223.log 2024-03-30T13:28:42.385 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.50683.log.gz 2024-03-30T13:28:42.385 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54135.log 2024-03-30T13:28:42.385 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.53848.log.gz 2024-03-30T13:28:42.385 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56688.log 2024-03-30T13:28:42.385 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.53223.log.gz 2024-03-30T13:28:42.385 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31000.log 2024-03-30T13:28:42.385 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54135.log.gz 2024-03-30T13:28:42.385 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48286.log 2024-03-30T13:28:42.386 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.56688.log.gz 2024-03-30T13:28:42.386 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48234.log 2024-03-30T13:28:42.386 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.31000.log.gz 2024-03-30T13:28:42.386 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54499.log 2024-03-30T13:28:42.386 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.48286.log.gz 2024-03-30T13:28:42.386 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35119.log 2024-03-30T13:28:42.386 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.48234.log.gz 2024-03-30T13:28:42.386 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43352.log 2024-03-30T13:28:42.386 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54499.log.gz 2024-03-30T13:28:42.386 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35119.log: 46.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.35119.log.gz 2024-03-30T13:28:42.386 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49067.log 2024-03-30T13:28:42.386 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56844.log 2024-03-30T13:28:42.386 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43352.log.gz 2024-03-30T13:28:42.386 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61859.log 2024-03-30T13:28:42.387 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49067.log.gz 2024-03-30T13:28:42.387 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45073.log 2024-03-30T13:28:42.387 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.56844.log.gz 2024-03-30T13:28:42.387 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45802.log 2024-03-30T13:28:42.387 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.61859.log.gz 2024-03-30T13:28:42.387 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44917.log 2024-03-30T13:28:42.387 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.45073.log.gz 2024-03-30T13:28:42.387 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58878.log 2024-03-30T13:28:42.387 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.45802.log.gz 2024-03-30T13:28:42.387 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31286.log 2024-03-30T13:28:42.387 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.44917.log.gz 2024-03-30T13:28:42.387 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52440.log 2024-03-30T13:28:42.387 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.58878.log.gz 2024-03-30T13:28:42.387 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38011.log 2024-03-30T13:28:42.387 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.31286.log.gz 2024-03-30T13:28:42.388 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43925.log 2024-03-30T13:28:42.388 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.52440.log.gz 2024-03-30T13:28:42.388 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43717.log 2024-03-30T13:28:42.388 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.38011.log.gz 2024-03-30T13:28:42.388 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62535.log 2024-03-30T13:28:42.388 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.43925.log.gz 2024-03-30T13:28:42.388 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61521.log 2024-03-30T13:28:42.388 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43717.log.gz 2024-03-30T13:28:42.388 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30479.log 2024-03-30T13:28:42.388 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.62535.log.gz 2024-03-30T13:28:42.388 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58826.log 2024-03-30T13:28:42.388 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.61521.log.gz 2024-03-30T13:28:42.388 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61339.log 2024-03-30T13:28:42.388 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.30479.log.gz 2024-03-30T13:28:42.388 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52622.log 2024-03-30T13:28:42.389 INFO:teuthology.orchestra.run.smithi067.stderr: 45.2% -- replaced with /var/log/ceph/ceph-client.admin.58826.log.gz 2024-03-30T13:28:42.389 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61339.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-30T13:28:42.389 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.61339.log.gz 2024-03-30T13:28:42.389 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54447.log 2024-03-30T13:28:42.389 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.52622.log.gz 2024-03-30T13:28:42.389 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39808.log 2024-03-30T13:28:42.389 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38999.log 2024-03-30T13:28:42.389 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54447.log.gz 2024-03-30T13:28:42.389 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64965.log 2024-03-30T13:28:42.389 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.39808.log.gz 2024-03-30T13:28:42.389 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56974.log 2024-03-30T13:28:42.389 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.38999.log.gz 2024-03-30T13:28:42.389 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64965.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.64965.log.gz 2024-03-30T13:28:42.390 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54317.log 2024-03-30T13:28:42.390 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64962.log 2024-03-30T13:28:42.390 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.56974.log.gz 2024-03-30T13:28:42.390 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46504.log 2024-03-30T13:28:42.390 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54317.log.gz 2024-03-30T13:28:42.390 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48625.log 2024-03-30T13:28:42.390 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64962.log.gz 2024-03-30T13:28:42.390 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46504.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.46504.log.gz 2024-03-30T13:28:42.390 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49223.log 2024-03-30T13:28:42.390 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31438.log 2024-03-30T13:28:42.390 INFO:teuthology.orchestra.run.smithi067.stderr: 47.7% -- replaced with /var/log/ceph/ceph-client.admin.48625.log.gz 2024-03-30T13:28:42.390 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.31802.log 2024-03-30T13:28:42.390 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.49223.log.gz 2024-03-30T13:28:42.390 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46036.log 2024-03-30T13:28:42.390 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.31438.log.gz 2024-03-30T13:28:42.391 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.0.31802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55228.log 2024-03-30T13:28:42.391 INFO:teuthology.orchestra.run.smithi067.stderr: 59.6% -- replaced with /var/log/ceph/ceph-client.0.31802.log.gz 2024-03-30T13:28:42.391 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49691.log 2024-03-30T13:28:42.391 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.46036.log.gz 2024-03-30T13:28:42.391 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60997.log 2024-03-30T13:28:42.391 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.55228.log.gz 2024-03-30T13:28:42.391 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49691.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49691.log.gz 2024-03-30T13:28:42.391 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35978.log 2024-03-30T13:28:42.391 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60919.log 2024-03-30T13:28:42.391 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.60997.log.gz 2024-03-30T13:28:42.391 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45489.log 2024-03-30T13:28:42.391 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.35978.log.gz 2024-03-30T13:28:42.391 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65562.log 2024-03-30T13:28:42.391 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.60919.log.gz 2024-03-30T13:28:42.391 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43509.log 46.7% 2024-03-30T13:28:42.391 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45489.log.gz 2024-03-30T13:28:42.392 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49665.log 2024-03-30T13:28:42.392 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.65562.log.gz 2024-03-30T13:28:42.392 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49353.log 2024-03-30T13:28:42.392 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43509.log.gz 2024-03-30T13:28:42.392 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36966.log 2024-03-30T13:28:42.392 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49665.log.gz 2024-03-30T13:28:42.392 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31889.log 2024-03-30T13:28:42.392 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49353.log.gz 2024-03-30T13:28:42.392 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36966.log: 46.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.36966.log.gz 2024-03-30T13:28:42.392 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.60971.log 2024-03-30T13:28:42.392 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59034.log 2024-03-30T13:28:42.392 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.31889.log.gz 2024-03-30T13:28:42.392 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41032.log 2024-03-30T13:28:42.392 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.60971.log.gz 2024-03-30T13:28:42.392 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49171.log 2024-03-30T13:28:42.393 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.59034.log.gz 2024-03-30T13:28:42.393 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41032.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.41032.log.gz 2024-03-30T13:28:42.393 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58930.log 2024-03-30T13:28:42.393 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44995.log 2024-03-30T13:28:42.393 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49171.log.gz 2024-03-30T13:28:42.393 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49955.log 2024-03-30T13:28:42.393 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.58930.log.gz 2024-03-30T13:28:42.393 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49015.log 2024-03-30T13:28:42.393 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.44995.log.gz 2024-03-30T13:28:42.393 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49955.log: gzip 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49955.log.gz -5 2024-03-30T13:28:42.393 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48208.log 2024-03-30T13:28:42.393 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59060.log 2024-03-30T13:28:42.393 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49015.log.gz 2024-03-30T13:28:42.393 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52937.log 2024-03-30T13:28:42.393 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.48208.log.gz 2024-03-30T13:28:42.394 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43405.log 2024-03-30T13:28:42.394 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.59060.log.gz 2024-03-30T13:28:42.394 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64626.log 2024-03-30T13:28:42.394 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.52937.log.gz 2024-03-30T13:28:42.394 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41710.log 2024-03-30T13:28:42.394 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43405.log.gz 2024-03-30T13:28:42.394 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53015.log 2024-03-30T13:28:42.394 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64626.log.gz 2024-03-30T13:28:42.394 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50059.log 2024-03-30T13:28:42.394 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.41710.log.gz 2024-03-30T13:28:42.394 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61547.log 2024-03-30T13:28:42.394 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.53015.log.gz 2024-03-30T13:28:42.394 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43665.log 2024-03-30T13:28:42.394 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.50059.log.gz 2024-03-30T13:28:42.394 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53301.log 2024-03-30T13:28:42.395 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.61547.log.gz 2024-03-30T13:28:42.395 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62613.log 2024-03-30T13:28:42.395 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43665.log.gz 2024-03-30T13:28:42.395 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53301.log: gzip -5 --verbose -- 46.7% /var/log/ceph/ceph-client.admin.61651.log 2024-03-30T13:28:42.395 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53301.log.gz 2024-03-30T13:28:42.395 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62613.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-30T13:28:42.395 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.62613.log.gz 2024-03-30T13:28:42.395 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49379.log 2024-03-30T13:28:42.395 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.61651.log.gz 2024-03-30T13:28:42.395 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63683.log 2024-03-30T13:28:42.395 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61781.log 2024-03-30T13:28:42.395 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49379.log.gz 2024-03-30T13:28:42.395 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60320.log 2024-03-30T13:28:42.395 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63683.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.63683.log.gz 2024-03-30T13:28:42.395 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47870.log 2024-03-30T13:28:42.396 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.61781.log.gz 2024-03-30T13:28:42.396 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45672.log 2024-03-30T13:28:42.396 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.60320.log.gz 2024-03-30T13:28:42.396 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51073.log 2024-03-30T13:28:42.396 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.47870.log.gz 2024-03-30T13:28:42.396 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36446.log 2024-03-30T13:28:42.396 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.45672.log.gz 2024-03-30T13:28:42.396 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32333.log 2024-03-30T13:28:42.396 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.51073.log.gz 2024-03-30T13:28:42.396 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36940.log 2024-03-30T13:28:42.396 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.36446.log.gz 2024-03-30T13:28:42.396 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31394.log 2024-03-30T13:28:42.396 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.32333.log.gz 2024-03-30T13:28:42.396 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64359.log 2024-03-30T13:28:42.396 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.36940.log.gz 2024-03-30T13:28:42.396 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42754.log 2024-03-30T13:28:42.397 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.31394.log.gz 2024-03-30T13:28:42.397 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60398.log 2024-03-30T13:28:42.397 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64359.log.gz 2024-03-30T13:28:42.397 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59982.log 2024-03-30T13:28:42.397 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.42754.log.gz 2024-03-30T13:28:42.397 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60398.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.60398.log.gz 2024-03-30T13:28:42.397 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32619.log 2024-03-30T13:28:42.397 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53171.log 2024-03-30T13:28:42.397 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.59982.log.gz 2024-03-30T13:28:42.397 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35405.log 2024-03-30T13:28:42.397 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.32619.log.gz 2024-03-30T13:28:42.397 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35588.log 2024-03-30T13:28:42.397 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53171.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.53171.log.gz 2024-03-30T13:28:42.397 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35405.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.35405.log.gz 2024-03-30T13:28:42.398 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32175.log 2024-03-30T13:28:42.398 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52021.log 2024-03-30T13:28:42.398 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35588.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.35588.log.gz 2024-03-30T13:28:42.398 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32175.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.32175.log.gz 2024-03-30T13:28:42.398 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30609.log 2024-03-30T13:28:42.398 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42728.log 2024-03-30T13:28:42.398 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52021.log: 47.7% -- replaced with /var/log/ceph/ceph-client.admin.52021.log.gz 2024-03-30T13:28:42.398 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41684.log 2024-03-30T13:28:42.398 INFO:teuthology.orchestra.run.smithi067.stderr: 44.9% -- replaced with /var/log/ceph/ceph-client.admin.30609.log.gz 2024-03-30T13:28:42.398 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46010.log 2024-03-30T13:28:42.398 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.42728.log.gz 2024-03-30T13:28:42.398 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35692.log 2024-03-30T13:28:42.398 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.41684.log.gz 2024-03-30T13:28:42.398 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46010.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.46010.log.gz 2024-03-30T13:28:42.398 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20206.log 2024-03-30T13:28:42.399 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50995.log 2024-03-30T13:28:42.399 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35692.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.35692.log.gz 2024-03-30T13:28:42.399 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph.tmp-client.admin.20206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55202.log 2024-03-30T13:28:42.399 INFO:teuthology.orchestra.run.smithi067.stderr: 66.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20206.log.gz 2024-03-30T13:28:42.399 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38609.log 2024-03-30T13:28:42.399 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.50995.log.gz 2024-03-30T13:28:42.399 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60814.log 2024-03-30T13:28:42.399 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.55202.log.gz 2024-03-30T13:28:42.399 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61179.log 2024-03-30T13:28:42.399 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.38609.log.gz 2024-03-30T13:28:42.399 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40252.log 2024-03-30T13:28:42.399 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.60814.log.gz 2024-03-30T13:28:42.399 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53874.log 2024-03-30T13:28:42.399 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.61179.log.gz 2024-03-30T13:28:42.399 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32489.log 2024-03-30T13:28:42.400 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.40252.log.gz 2024-03-30T13:28:42.400 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46140.log 2024-03-30T13:28:42.400 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.53874.log.gz 2024-03-30T13:28:42.400 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65744.log 2024-03-30T13:28:42.400 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.32489.log.gz 2024-03-30T13:28:42.400 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32097.log 2024-03-30T13:28:42.400 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.46140.log.gz 2024-03-30T13:28:42.400 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49093.log 2024-03-30T13:28:42.401 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.65744.log.gz 2024-03-30T13:28:42.401 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64496.log 2024-03-30T13:28:42.401 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.32097.log.gz 2024-03-30T13:28:42.401 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41658.log 2024-03-30T13:28:42.401 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.49093.log.gz 2024-03-30T13:28:42.401 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64496.log: 46.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.64496.log.gz 2024-03-30T13:28:42.401 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.60164.log 2024-03-30T13:28:42.401 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33191.log 2024-03-30T13:28:42.401 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41658.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.41658.log.gz 2024-03-30T13:28:42.401 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65978.log 2024-03-30T13:28:42.401 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.60164.log.gz 2024-03-30T13:28:42.401 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60008.log 2024-03-30T13:28:42.401 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.33191.log.gz 2024-03-30T13:28:42.401 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36758.log 2024-03-30T13:28:42.401 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.65978.log.gz 2024-03-30T13:28:42.402 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65154.log 2024-03-30T13:28:42.402 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.60008.log.gz 2024-03-30T13:28:42.402 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36290.log 2024-03-30T13:28:42.402 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36758.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.36758.log.gz 2024-03-30T13:28:42.402 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45281.log 2024-03-30T13:28:42.402 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.65154.log.gz 2024-03-30T13:28:42.402 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35249.log 2024-03-30T13:28:42.402 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.36290.log.gz 2024-03-30T13:28:42.402 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54838.log 2024-03-30T13:28:42.402 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45281.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.45281.log.gz 2024-03-30T13:28:42.402 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42806.log 2024-03-30T13:28:42.402 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.35249.log.gz 2024-03-30T13:28:42.402 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41190.log 2024-03-30T13:28:42.402 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54838.log.gz 2024-03-30T13:28:42.402 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45021.log 2024-03-30T13:28:42.403 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42806.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.42806.log.gz 2024-03-30T13:28:42.403 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40122.log 2024-03-30T13:28:42.403 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.41190.log.gz 2024-03-30T13:28:42.403 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31431.log 2024-03-30T13:28:42.403 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.45021.log.gz 2024-03-30T13:28:42.403 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32149.log 2024-03-30T13:28:42.403 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40122.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.40122.log.gz 2024-03-30T13:28:42.403 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59008.log 2024-03-30T13:28:42.403 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.31431.log.gz 2024-03-30T13:28:42.403 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34703.log 2024-03-30T13:28:42.403 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.32149.log.gz 2024-03-30T13:28:42.403 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61755.log 2024-03-30T13:28:42.403 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59008.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.59008.log.gz 2024-03-30T13:28:42.403 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32411.log 2024-03-30T13:28:42.403 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.34703.log.gz 2024-03-30T13:28:42.404 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30610.log 2024-03-30T13:28:42.404 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.61755.log.gz 2024-03-30T13:28:42.404 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45750.log 2024-03-30T13:28:42.404 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32411.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.32411.log.gz 2024-03-30T13:28:42.404 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43274.log 2024-03-30T13:28:42.404 INFO:teuthology.orchestra.run.smithi067.stderr: 44.9% -- replaced with /var/log/ceph/ceph-client.admin.30610.log.gz 2024-03-30T13:28:42.404 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37307.log 2024-03-30T13:28:42.404 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.45750.log.gz 2024-03-30T13:28:42.404 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40954.log 2024-03-30T13:28:42.404 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43274.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43274.log.gz 2024-03-30T13:28:42.404 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32463.log 2024-03-30T13:28:42.404 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.37307.log.gz 2024-03-30T13:28:42.404 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65168.log 2024-03-30T13:28:42.404 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.40954.log.gz 2024-03-30T13:28:42.404 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46062.log 2024-03-30T13:28:42.404 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32463.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.32463.log.gz 2024-03-30T13:28:42.405 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65168.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40538.log 2024-03-30T13:28:42.405 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.65168.log.gz 2024-03-30T13:28:42.405 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63995.log 2024-03-30T13:28:42.405 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.46062.log.gz 2024-03-30T13:28:42.405 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43821.log 2024-03-30T13:28:42.405 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40538.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.40538.log.gz 2024-03-30T13:28:42.405 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41398.log 2024-03-30T13:28:42.405 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.63995.log.gz 2024-03-30T13:28:42.405 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37803.log 2024-03-30T13:28:42.405 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43821.log.gz 2024-03-30T13:28:42.405 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35535.log 2024-03-30T13:28:42.405 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41398.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.41398.log.gz 2024-03-30T13:28:42.405 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52570.log 2024-03-30T13:28:42.405 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.37803.log.gz 2024-03-30T13:28:42.405 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62587.log 2024-03-30T13:28:42.406 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.35535.log.gz 2024-03-30T13:28:42.406 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47974.log 2024-03-30T13:28:42.406 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52570.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.52570.log.gz 2024-03-30T13:28:42.406 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34521.log 2024-03-30T13:28:42.406 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.62587.log.gz 2024-03-30T13:28:42.406 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32045.log 2024-03-30T13:28:42.406 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.47974.log.gz 2024-03-30T13:28:42.406 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53354.log 2024-03-30T13:28:42.406 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34521.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.34521.log.gz 2024-03-30T13:28:42.406 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46582.log 2024-03-30T13:28:42.406 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.32045.log.gz 2024-03-30T13:28:42.406 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65796.log 2024-03-30T13:28:42.406 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.53354.log.gz 2024-03-30T13:28:42.406 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66004.log 2024-03-30T13:28:42.406 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46582.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.46582.log.gz 2024-03-30T13:28:42.410 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33373.log 2024-03-30T13:28:42.410 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.65796.log.gz 2024-03-30T13:28:42.410 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.66004.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.37697.log 2024-03-30T13:28:42.410 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.66004.log.gz 2024-03-30T13:28:42.410 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60060.log 2024-03-30T13:28:42.410 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.33373.log.gz 2024-03-30T13:28:42.410 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60736.log 2024-03-30T13:28:42.410 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.37697.log.gz 2024-03-30T13:28:42.410 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30532.log 2024-03-30T13:28:42.411 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.60060.log.gz 2024-03-30T13:28:42.411 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64961.log 2024-03-30T13:28:42.411 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60736.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.60736.log.gz 2024-03-30T13:28:42.411 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47818.log 2024-03-30T13:28:42.411 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.30532.log.gz 2024-03-30T13:28:42.411 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31941.log 2024-03-30T13:28:42.411 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.64961.log.gz 2024-03-30T13:28:42.411 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30611.log 2024-03-30T13:28:42.411 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47818.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.47818.log.gz 2024-03-30T13:28:42.411 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60606.log 2024-03-30T13:28:42.411 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.31941.log.gz 2024-03-30T13:28:42.411 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43300.log 2024-03-30T13:28:42.411 INFO:teuthology.orchestra.run.smithi067.stderr: 44.9% -- replaced with /var/log/ceph/ceph-client.admin.30611.log.gz 2024-03-30T13:28:42.411 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37125.log 2024-03-30T13:28:42.411 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60606.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.60606.log.gz 2024-03-30T13:28:42.412 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39366.log 2024-03-30T13:28:42.412 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43300.log.gz 2024-03-30T13:28:42.412 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36654.log 2024-03-30T13:28:42.412 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.37125.log.gz 2024-03-30T13:28:42.412 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65379.log/var/log/ceph/ceph-client.admin.39366.log: 2024-03-30T13:28:42.412 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.39366.log.gz 2024-03-30T13:28:42.412 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45437.log 2024-03-30T13:28:42.412 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.36654.log.gz 2024-03-30T13:28:42.412 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54109.log 2024-03-30T13:28:42.412 INFO:teuthology.orchestra.run.smithi067.stderr: 44.4% -- replaced with /var/log/ceph/ceph-client.admin.65379.log.gz 2024-03-30T13:28:42.412 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34599.log 2024-03-30T13:28:42.412 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.45437.log.gz 2024-03-30T13:28:42.412 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61101.log 2024-03-30T13:28:42.412 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54109.log.gz 2024-03-30T13:28:42.412 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32567.log 2024-03-30T13:28:42.413 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.34599.log.gz 2024-03-30T13:28:42.413 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65614.log 2024-03-30T13:28:42.413 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61101.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.61101.log.gz 2024-03-30T13:28:42.413 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52752.log 2024-03-30T13:28:42.413 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.32567.log.gz 2024-03-30T13:28:42.413 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34989.log 2024-03-30T13:28:42.413 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.65614.log.gz 2024-03-30T13:28:42.413 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56870.log 2024-03-30T13:28:42.413 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52752.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.52752.log.gz 2024-03-30T13:28:42.413 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52125.log 2024-03-30T13:28:42.413 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.34989.log.gz 2024-03-30T13:28:42.413 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60242.log 2024-03-30T13:28:42.413 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.56870.log.gz 2024-03-30T13:28:42.413 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32019.log 2024-03-30T13:28:42.413 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.52125.log.gz 2024-03-30T13:28:42.414 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31026.log 2024-03-30T13:28:42.414 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.60242.log.gz 2024-03-30T13:28:42.414 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49041.log 2024-03-30T13:28:42.414 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.32019.log.gz 2024-03-30T13:28:42.414 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61231.log 2024-03-30T13:28:42.414 INFO:teuthology.orchestra.run.smithi067.stderr: 45.8% -- replaced with /var/log/ceph/ceph-client.admin.31026.log.gz 2024-03-30T13:28:42.414 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53432.log 2024-03-30T13:28:42.414 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.49041.log.gz 2024-03-30T13:28:42.414 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62561.log 2024-03-30T13:28:42.414 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.61231.log.gz 2024-03-30T13:28:42.414 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65354.log 2024-03-30T13:28:42.414 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53432.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.53432.log.gz 2024-03-30T13:28:42.414 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63865.log 2024-03-30T13:28:42.414 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.62561.log.gz 2024-03-30T13:28:42.414 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64966.log 2024-03-30T13:28:42.415 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.65354.log.gz 2024-03-30T13:28:42.415 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5/var/log/ceph/ceph-client.admin.63865.log: --verbose -- /var/log/ceph/ceph-client.admin.44969.log 2024-03-30T13:28:42.415 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.63865.log.gz 2024-03-30T13:28:42.415 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41346.log 2024-03-30T13:28:42.415 INFO:teuthology.orchestra.run.smithi067.stderr: 44.4% -- replaced with /var/log/ceph/ceph-client.admin.64966.log.gz 2024-03-30T13:28:42.415 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39129.log 2024-03-30T13:28:42.415 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44969.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.44969.log.gz 2024-03-30T13:28:42.415 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31539.log 2024-03-30T13:28:42.415 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.41346.log.gz 2024-03-30T13:28:42.415 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39548.log 2024-03-30T13:28:42.415 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.39129.log.gz 2024-03-30T13:28:42.415 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35431.log 2024-03-30T13:28:42.415 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.31539.log.gz 2024-03-30T13:28:42.415 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50787.log 2024-03-30T13:28:42.415 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.39548.log.gz 2024-03-30T13:28:42.416 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61049.log 2024-03-30T13:28:42.416 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.35431.log.gz 2024-03-30T13:28:42.416 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56402.log 2024-03-30T13:28:42.416 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.50787.log.gz 2024-03-30T13:28:42.416 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43535.log 2024-03-30T13:28:42.416 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.61049.log.gz 2024-03-30T13:28:42.416 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31689.log 2024-03-30T13:28:42.416 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.56402.log.gz 2024-03-30T13:28:42.416 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52780.log 2024-03-30T13:28:42.416 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43535.log.gz 2024-03-30T13:28:42.416 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.31689.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35666.log 2024-03-30T13:28:42.416 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.31689.log.gz 2024-03-30T13:28:42.416 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61127.log 2024-03-30T13:28:42.416 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.52780.log.gz 2024-03-30T13:28:42.416 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34807.log 2024-03-30T13:28:42.416 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.35666.log.gz 2024-03-30T13:28:42.417 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54473.log 2024-03-30T13:28:42.417 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61127.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.61127.log.gz 2024-03-30T13:28:42.417 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50397.log 2024-03-30T13:28:42.417 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.34807.log.gz 2024-03-30T13:28:42.417 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58670.log 2024-03-30T13:28:42.417 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54473.log.gz 2024-03-30T13:28:42.417 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56818.log 2024-03-30T13:28:42.417 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.50397.log.gz 2024-03-30T13:28:42.417 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46166.log 2024-03-30T13:28:42.417 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.58670.log.gz 2024-03-30T13:28:42.417 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43196.log 2024-03-30T13:28:42.417 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.56818.log.gz 2024-03-30T13:28:42.417 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59956.log 2024-03-30T13:28:42.417 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46166.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.46166.log.gz 2024-03-30T13:28:42.417 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37018.log 2024-03-30T13:28:42.418 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43196.log.gz 2024-03-30T13:28:42.418 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58592.log 2024-03-30T13:28:42.418 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.59956.log.gz 2024-03-30T13:28:42.418 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40200.log/var/log/ceph/ceph-client.admin.37018.log: 2024-03-30T13:28:42.418 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.37018.log.gz 2024-03-30T13:28:42.418 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58592.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-30T13:28:42.418 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.58592.log.gz 2024-03-30T13:28:42.418 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40278.log 2024-03-30T13:28:42.418 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.40200.log.gz 2024-03-30T13:28:42.418 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose --/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.45229.log 2024-03-30T13:28:42.418 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64964.log 2024-03-30T13:28:42.418 INFO:teuthology.orchestra.run.smithi067.stderr: 45.2% -- replaced with /var/log/ceph/ceph-client.admin.40278.log.gz 2024-03-30T13:28:42.418 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50813.log 2024-03-30T13:28:42.418 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.45229.log.gz 2024-03-30T13:28:42.418 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38843.log 2024-03-30T13:28:42.419 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64964.log.gz 2024-03-30T13:28:42.419 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52518.log 2024-03-30T13:28:42.419 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.50813.log.gz 2024-03-30T13:28:42.419 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43899.log 2024-03-30T13:28:42.419 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.38843.log.gz 2024-03-30T13:28:42.419 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38583.log 2024-03-30T13:28:42.419 INFO:teuthology.orchestra.run.smithi067.stderr: 44.9% -- replaced with /var/log/ceph/ceph-client.admin.52518.log.gz 2024-03-30T13:28:42.419 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39940.log 2024-03-30T13:28:42.419 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43899.log.gz 2024-03-30T13:28:42.419 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39236.log 2024-03-30T13:28:42.419 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.38583.log.gz 2024-03-30T13:28:42.419 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57442.log 2024-03-30T13:28:42.419 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.39940.log.gz 2024-03-30T13:28:42.419 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55358.log 2024-03-30T13:28:42.419 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.39236.log.gz 2024-03-30T13:28:42.420 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56480.log 2024-03-30T13:28:42.420 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.57442.log.gz 2024-03-30T13:28:42.420 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50969.log 2024-03-30T13:28:42.420 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.55358.log.gz 2024-03-30T13:28:42.420 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43691.log 2024-03-30T13:28:42.420 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.56480.log.gz 2024-03-30T13:28:42.420 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41554.log 2024-03-30T13:28:42.420 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.50969.log.gz 2024-03-30T13:28:42.420 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38297.log 2024-03-30T13:28:42.420 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43691.log.gz 2024-03-30T13:28:42.420 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36030.log 2024-03-30T13:28:42.420 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.41554.log.gz 2024-03-30T13:28:42.420 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34937.log 2024-03-30T13:28:42.420 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.38297.log.gz 2024-03-30T13:28:42.420 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38505.log 2024-03-30T13:28:42.421 INFO:teuthology.orchestra.run.smithi067.stderr: 47.7% -- replaced with /var/log/ceph/ceph-client.admin.36030.log.gz 2024-03-30T13:28:42.421 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58410.log 2024-03-30T13:28:42.421 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.34937.log.gz 2024-03-30T13:28:42.421 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61417.log 2024-03-30T13:28:42.421 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.38505.log.gz 2024-03-30T13:28:42.421 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43118.log 2024-03-30T13:28:42.421 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.58410.log.gz 2024-03-30T13:28:42.421 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53588.log 2024-03-30T13:28:42.421 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.61417.log.gz 2024-03-30T13:28:42.421 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40798.log 2024-03-30T13:28:42.421 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43118.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43118.log.gz 2024-03-30T13:28:42.421 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35848.log 2024-03-30T13:28:42.421 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.53588.log.gz 2024-03-30T13:28:42.421 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25331.log 2024-03-30T13:28:42.421 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.40798.log.gz 2024-03-30T13:28:42.421 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33217.log 2024-03-30T13:28:42.422 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.35848.log.gz 2024-03-30T13:28:42.422 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.25331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38869.log 2024-03-30T13:28:42.422 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.25331.log.gz 2024-03-30T13:28:42.422 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45932.log 2024-03-30T13:28:42.422 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.33217.log.gz 2024-03-30T13:28:42.422 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39392.log 2024-03-30T13:28:42.422 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.38869.log.gz 2024-03-30T13:28:42.422 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65161.log 2024-03-30T13:28:42.422 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.45932.log.gz 2024-03-30T13:28:42.422 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50865.log 2024-03-30T13:28:42.422 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39392.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.39392.log.gz 2024-03-30T13:28:42.422 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48390.log 2024-03-30T13:28:42.422 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.65161.log.gz 2024-03-30T13:28:42.422 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60502.log 2024-03-30T13:28:42.422 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.50865.log.gz 2024-03-30T13:28:42.423 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32957.log 2024-03-30T13:28:42.423 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48390.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.48390.log.gz 2024-03-30T13:28:42.423 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31011.log 2024-03-30T13:28:42.423 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.60502.log.gz 2024-03-30T13:28:42.423 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40434.log 2024-03-30T13:28:42.423 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.32957.log.gz 2024-03-30T13:28:42.423 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34729.log 2024-03-30T13:28:42.423 INFO:teuthology.orchestra.run.smithi067.stderr: 44.9% -- replaced with /var/log/ceph/ceph-client.admin.31011.log.gz 2024-03-30T13:28:42.423 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48078.log 2024-03-30T13:28:42.423 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.40434.log.gz 2024-03-30T13:28:42.423 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45593.log 2024-03-30T13:28:42.423 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.34729.log.gz 2024-03-30T13:28:42.423 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53718.log 2024-03-30T13:28:42.423 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.48078.log.gz 2024-03-30T13:28:42.423 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64468.log 2024-03-30T13:28:42.424 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.45593.log.gz 2024-03-30T13:28:42.424 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48599.log 2024-03-30T13:28:42.424 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53718.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.53718.log.gz 2024-03-30T13:28:42.424 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41528.log 2024-03-30T13:28:42.424 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64468.log.gz 2024-03-30T13:28:42.424 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42884.log 2024-03-30T13:28:42.424 INFO:teuthology.orchestra.run.smithi067.stderr: 48.1% -- replaced with /var/log/ceph/ceph-client.admin.48599.log.gz 2024-03-30T13:28:42.424 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.41528.log: -- /var/log/ceph/ceph-client.admin.53640.log 2024-03-30T13:28:42.424 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.41528.log.gz 2024-03-30T13:28:42.424 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.42884.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.63917.log 2024-03-30T13:28:42.424 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.42884.log.gz 2024-03-30T13:28:42.424 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57052.log 2024-03-30T13:28:42.424 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.53640.log.gz 2024-03-30T13:28:42.424 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35640.log 2024-03-30T13:28:42.424 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.63917.log.gz 2024-03-30T13:28:42.424 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45698.log 2024-03-30T13:28:42.425 INFO:teuthology.orchestra.run.smithi067.stderr: 47.7% -- replaced with /var/log/ceph/ceph-client.admin.57052.log.gz 2024-03-30T13:28:42.425 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41580.log 2024-03-30T13:28:42.425 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.35640.log.gz 2024-03-30T13:28:42.425 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.45698.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.58722.log 2024-03-30T13:28:42.425 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.45698.log.gz 2024-03-30T13:28:42.425 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40720.log 2024-03-30T13:28:42.425 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.41580.log.gz 2024-03-30T13:28:42.425 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36186.log 2024-03-30T13:28:42.425 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.58722.log.gz 2024-03-30T13:28:42.425 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42780.log 2024-03-30T13:28:42.425 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40720.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.40720.log.gz 2024-03-30T13:28:42.425 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38089.log 2024-03-30T13:28:42.425 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.36186.log.gz 2024-03-30T13:28:42.425 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45333.log 2024-03-30T13:28:42.425 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.42780.log.gz 2024-03-30T13:28:42.426 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43170.log 2024-03-30T13:28:42.426 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38089.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.38089.log.gz 2024-03-30T13:28:42.539 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45385.log 2024-03-30T13:28:42.539 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.45333.log.gz 2024-03-30T13:28:42.539 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65361.log 2024-03-30T13:28:42.539 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43170.log.gz 2024-03-30T13:28:42.539 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40668.log 2024-03-30T13:28:42.539 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45385.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.45385.log.gz 2024-03-30T13:28:42.539 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53692.log 2024-03-30T13:28:42.539 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.65361.log.gz 2024-03-30T13:28:42.539 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64099.log 2024-03-30T13:28:42.539 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.40668.log.gz 2024-03-30T13:28:42.539 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51995.log 2024-03-30T13:28:42.540 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.53692.log.gz 2024-03-30T13:28:42.540 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53536.log 2024-03-30T13:28:42.540 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.64099.log.gz 2024-03-30T13:28:42.540 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31993.log 2024-03-30T13:28:42.540 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.51995.log.gz 2024-03-30T13:28:42.540 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30612.log 2024-03-30T13:28:42.540 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.53536.log.gz 2024-03-30T13:28:42.540 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46114.log 2024-03-30T13:28:42.540 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.31993.log.gz 2024-03-30T13:28:42.540 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32905.log 2024-03-30T13:28:42.540 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.30612.log.gz 2024-03-30T13:28:42.540 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60268.log 2024-03-30T13:28:42.540 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38323.log 2024-03-30T13:28:42.540 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.32905.log.gz 2024-03-30T13:28:42.540 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.46114.log.gz 2024-03-30T13:28:42.541 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60866.log 2024-03-30T13:28:42.541 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.60268.log.gz 2024-03-30T13:28:42.541 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64957.log 2024-03-30T13:28:42.541 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.38323.log.gz 2024-03-30T13:28:42.541 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33451.log 2024-03-30T13:28:42.541 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.60866.log.gz 2024-03-30T13:28:42.541 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34677.log 2024-03-30T13:28:42.541 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64957.log.gz 2024-03-30T13:28:42.541 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43639.log 2024-03-30T13:28:42.541 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.33451.log.gz 2024-03-30T13:28:42.541 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36862.log 2024-03-30T13:28:42.541 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34677.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.34677.log.gz 2024-03-30T13:28:42.541 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31208.log 2024-03-30T13:28:42.541 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43639.log.gz 2024-03-30T13:28:42.541 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64125.log 2024-03-30T13:28:42.541 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.36862.log.gz 2024-03-30T13:28:42.542 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61443.log 2024-03-30T13:28:42.542 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.31208.log.gz 2024-03-30T13:28:42.542 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62093.log 2024-03-30T13:28:42.542 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64125.log.gz 2024-03-30T13:28:42.542 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41814.log 2024-03-30T13:28:42.542 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61443.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.61443.log.gz 2024-03-30T13:28:42.542 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48312.log 2024-03-30T13:28:42.542 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.62093.log.gz 2024-03-30T13:28:42.542 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61023.log 2024-03-30T13:28:42.542 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.41814.log.gz 2024-03-30T13:28:42.542 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48651.log 2024-03-30T13:28:42.542 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.48312.log.gz 2024-03-30T13:28:42.542 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37541.log 2024-03-30T13:28:42.542 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.61023.log.gz 2024-03-30T13:28:42.543 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52307.log 2024-03-30T13:28:42.543 INFO:teuthology.orchestra.run.smithi067.stderr: 47.7% -- replaced with /var/log/ceph/ceph-client.admin.48651.log.gz 2024-03-30T13:28:42.543 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37541.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-30T13:28:42.543 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.37541.log.gz 2024-03-30T13:28:42.543 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53328.log 2024-03-30T13:28:42.543 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.52307.log.gz 2024-03-30T13:28:42.543 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53484.log 2024-03-30T13:28:42.543 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55150.log 2024-03-30T13:28:42.543 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.53328.log.gz 2024-03-30T13:28:42.543 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52648.log 2024-03-30T13:28:42.543 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.53484.log.gz 2024-03-30T13:28:42.543 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60684.log 2024-03-30T13:28:42.543 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55150.log: /var/log/ceph/ceph-client.admin.52648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45984.log 2024-03-30T13:28:42.543 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.55150.log.gz 2024-03-30T13:28:42.543 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.52648.log.gz/var/log/ceph/ceph-client.admin.60684.log: 2024-03-30T13:28:42.543 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.60684.log.gz 2024-03-30T13:28:42.544 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53952.log 2024-03-30T13:28:42.544 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51125.log 2024-03-30T13:28:42.544 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45984.log: /var/log/ceph/ceph-client.admin.53952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35275.log 2024-03-30T13:28:42.544 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.45984.log.gz 2024-03-30T13:28:42.544 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.53952.log.gz/var/log/ceph/ceph-client.admin.51125.log: 2024-03-30T13:28:42.544 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39992.log 2024-03-30T13:28:42.544 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.51125.log.gz 2024-03-30T13:28:42.544 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30599.log 2024-03-30T13:28:42.544 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35275.log: /var/log/ceph/ceph-client.admin.39992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50579.log 2024-03-30T13:28:42.544 INFO:teuthology.orchestra.run.smithi067.stderr: 48.1% -- replaced with /var/log/ceph/ceph-client.admin.35275.log.gz 2024-03-30T13:28:42.544 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.39992.log.gz/var/log/ceph/ceph-client.admin.30599.log: 2024-03-30T13:28:42.544 INFO:teuthology.orchestra.run.smithi067.stderr: 47.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.30599.log.gz 2024-03-30T13:28:42.544 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.58774.log 2024-03-30T13:28:42.544 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48260.log 2024-03-30T13:28:42.544 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50579.log: /var/log/ceph/ceph-client.admin.58774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35197.log 2024-03-30T13:28:42.545 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.50579.log.gz 2024-03-30T13:28:42.545 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.58774.log.gz/var/log/ceph/ceph-client.admin.48260.log: 2024-03-30T13:28:42.545 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.48260.log.gz 2024-03-30T13:28:42.545 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30603.log 2024-03-30T13:28:42.545 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37985.log 2024-03-30T13:28:42.545 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35197.log: /var/log/ceph/ceph-client.admin.30603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48755.log 2024-03-30T13:28:42.545 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% 46.7% -- replaced with /var/log/ceph/ceph-client.admin.35197.log.gz 2024-03-30T13:28:42.545 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30603.log.gz 2024-03-30T13:28:42.545 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37985.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.37985.log.gz 2024-03-30T13:28:42.545 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50033.log 2024-03-30T13:28:42.545 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44055.log 2024-03-30T13:28:42.545 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.48755.log.gz 2024-03-30T13:28:42.545 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42910.log 2024-03-30T13:28:42.545 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.50033.log.gz 2024-03-30T13:28:42.545 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65146.log 2024-03-30T13:28:42.546 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44055.log: /var/log/ceph/ceph-client.admin.42910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44943.log 2024-03-30T13:28:42.546 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% 47.2% -- replaced with /var/log/ceph/ceph-client.admin.44055.log.gz 2024-03-30T13:28:42.546 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42910.log.gz 2024-03-30T13:28:42.546 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65146.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.65146.log.gz 2024-03-30T13:28:42.546 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45567.log 2024-03-30T13:28:42.546 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62457.log 2024-03-30T13:28:42.546 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44943.log: /var/log/ceph/ceph-client.admin.45567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45828.log 2024-03-30T13:28:42.546 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.44943.log.gz 2024-03-30T13:28:42.546 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.45567.log.gz 2024-03-30T13:28:42.546 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62457.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.62457.log.gz 2024-03-30T13:28:42.546 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61365.log 2024-03-30T13:28:42.546 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36784.log 2024-03-30T13:28:42.546 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45828.log: /var/log/ceph/ceph-client.admin.61365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43326.log 2024-03-30T13:28:42.546 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.45828.log.gz 2024-03-30T13:28:42.546 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3%/var/log/ceph/ceph-client.admin.36784.log: -- replaced with /var/log/ceph/ceph-client.admin.61365.log.gz 2024-03-30T13:28:42.547 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.36784.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57208.log 2024-03-30T13:28:42.547 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.547 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41162.log 2024-03-30T13:28:42.547 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43326.log: /var/log/ceph/ceph-client.admin.57208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40356.log 2024-03-30T13:28:42.547 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43326.log.gz 2024-03-30T13:28:42.547 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.57208.log.gz/var/log/ceph/ceph-client.admin.41162.log: 2024-03-30T13:28:42.547 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37829.log 2024-03-30T13:28:42.547 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41162.log.gz 2024-03-30T13:28:42.547 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40356.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.40356.log.gz 2024-03-30T13:28:42.547 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35301.log 2024-03-30T13:28:42.547 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57286.log 2024-03-30T13:28:42.547 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37829.log: /var/log/ceph/ceph-client.admin.35301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31010.log 2024-03-30T13:28:42.547 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.37829.log.gz 2024-03-30T13:28:42.547 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.35301.log.gz 2024-03-30T13:28:42.547 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57286.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.57286.log.gz 2024-03-30T13:28:42.548 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39862.log 2024-03-30T13:28:42.548 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54213.log 2024-03-30T13:28:42.548 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31010.log: /var/log/ceph/ceph-client.admin.39862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65170.log 2024-03-30T13:28:42.548 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% 46.7% -- replaced with /var/log/ceph/ceph-client.admin.39862.log.gz 2024-03-30T13:28:42.548 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31010.log.gz 2024-03-30T13:28:42.548 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54213.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54213.log.gz 2024-03-30T13:28:42.548 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57234.log 2024-03-30T13:28:42.548 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38453.log 2024-03-30T13:28:42.548 INFO:teuthology.orchestra.run.smithi067.stderr: 43.9% -- replaced with /var/log/ceph/ceph-client.admin.65170.log.gz 2024-03-30T13:28:42.548 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61075.log 2024-03-30T13:28:42.548 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.57234.log.gz 2024-03-30T13:28:42.548 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38453.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.38453.log.gz 2024-03-30T13:28:42.548 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36420.log 2024-03-30T13:28:42.548 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35041.log 2024-03-30T13:28:42.548 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61075.log: /var/log/ceph/ceph-client.admin.36420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36342.log 2024-03-30T13:28:42.548 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.61075.log.gz 2024-03-30T13:28:42.549 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.36420.log.gz 2024-03-30T13:28:42.549 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35041.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.35041.log.gz 2024-03-30T13:28:42.549 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46686.log 2024-03-30T13:28:42.549 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62301.log 2024-03-30T13:28:42.549 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36342.log: /var/log/ceph/ceph-client.admin.46686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58566.log 2024-03-30T13:28:42.549 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% 46.7% -- replaced with /var/log/ceph/ceph-client.admin.46686.log.gz 2024-03-30T13:28:42.549 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36342.log.gz 2024-03-30T13:28:42.549 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62301.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.62301.log.gz 2024-03-30T13:28:42.549 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39756.log 2024-03-30T13:28:42.549 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54239.log 2024-03-30T13:28:42.549 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.58566.log.gz 2024-03-30T13:28:42.550 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45776.log 2024-03-30T13:28:42.550 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.39756.log.gz 2024-03-30T13:28:42.550 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40590.log 2024-03-30T13:28:42.550 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54239.log: /var/log/ceph/ceph-client.admin.45776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55176.log 2024-03-30T13:28:42.550 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54239.log.gz 2024-03-30T13:28:42.550 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.45776.log.gz/var/log/ceph/ceph-client.admin.40590.log: 2024-03-30T13:28:42.550 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49851.log 2024-03-30T13:28:42.550 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40590.log.gz 2024-03-30T13:28:42.550 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55176.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.55176.log.gz 2024-03-30T13:28:42.550 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45151.log 2024-03-30T13:28:42.550 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64437.log 2024-03-30T13:28:42.550 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49851.log.gz 2024-03-30T13:28:42.550 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33555.log 2024-03-30T13:28:42.551 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.45151.log.gz 2024-03-30T13:28:42.551 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63579.log 2024-03-30T13:28:42.551 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64437.log: /var/log/ceph/ceph-client.admin.33555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64600.log 2024-03-30T13:28:42.551 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64437.log.gz 2024-03-30T13:28:42.551 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.33555.log.gz 2024-03-30T13:28:42.551 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63579.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.63579.log.gz 2024-03-30T13:28:42.551 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63969.log 2024-03-30T13:28:42.551 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38817.log 2024-03-30T13:28:42.551 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64600.log.gz 2024-03-30T13:28:42.551 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60138.log 2024-03-30T13:28:42.551 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.63969.log.gz 2024-03-30T13:28:42.551 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36238.log 2024-03-30T13:28:42.551 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38817.log: gzip/var/log/ceph/ceph-client.admin.60138.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40304.log 2024-03-30T13:28:42.551 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.60138.log.gz 2024-03-30T13:28:42.551 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.38817.log.gz 2024-03-30T13:28:42.551 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36238.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.36238.log.gz 2024-03-30T13:28:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36212.log 2024-03-30T13:28:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41216.log 2024-03-30T13:28:42.552 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.40304.log.gz 2024-03-30T13:28:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54603.log 2024-03-30T13:28:42.552 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.36212.log.gz 2024-03-30T13:28:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52492.log 2024-03-30T13:28:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41216.log: /var/log/ceph/ceph-client.admin.54603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31636.log 2024-03-30T13:28:42.552 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54603.log.gz 2024-03-30T13:28:42.552 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41216.log.gz 2024-03-30T13:28:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52492.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.52492.log.gz 2024-03-30T13:28:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64151.log 2024-03-30T13:28:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50111.log 2024-03-30T13:28:42.552 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.31636.log.gz 2024-03-30T13:28:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35223.log 2024-03-30T13:28:42.552 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64151.log.gz 2024-03-30T13:28:42.553 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60086.log 2024-03-30T13:28:42.553 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50111.log: /var/log/ceph/ceph-client.admin.35223.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.50111.log.gz 2024-03-30T13:28:42.553 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.35223.log.gz 2024-03-30T13:28:42.553 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46634.log 2024-03-30T13:28:42.553 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38375.log 2024-03-30T13:28:42.553 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60086.log: /var/log/ceph/ceph-client.admin.46634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33087.log 2024-03-30T13:28:42.553 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.60086.log.gz 2024-03-30T13:28:42.553 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.46634.log.gz/var/log/ceph/ceph-client.admin.38375.log: 2024-03-30T13:28:42.553 INFO:teuthology.orchestra.run.smithi067.stderr: 45.2% -- replaced with /var/log/ceph/ceph-client.admin.38375.log.gz 2024-03-30T13:28:42.553 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53562.log 2024-03-30T13:28:42.553 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36264.log 2024-03-30T13:28:42.553 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33087.log: /var/log/ceph/ceph-client.admin.53562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38531.log 2024-03-30T13:28:42.553 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.33087.log.gz 2024-03-30T13:28:42.553 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.53562.log.gz 2024-03-30T13:28:42.553 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36264.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.36264.log.gz 2024-03-30T13:28:42.553 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52544.log 2024-03-30T13:28:42.554 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57520.log 2024-03-30T13:28:42.554 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38531.log: /var/log/ceph/ceph-client.admin.52544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65174.log 2024-03-30T13:28:42.554 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.38531.log.gz 2024-03-30T13:28:42.554 INFO:teuthology.orchestra.run.smithi067.stderr: 47.7% -- replaced with /var/log/ceph/ceph-client.admin.52544.log.gz/var/log/ceph/ceph-client.admin.57520.log: 2024-03-30T13:28:42.554 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.57520.log.gz 2024-03-30T13:28:42.554 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36316.log 2024-03-30T13:28:42.554 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61625.log 2024-03-30T13:28:42.554 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65174.log: /var/log/ceph/ceph-client.admin.36316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43014.log 2024-03-30T13:28:42.554 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.65174.log.gz 2024-03-30T13:28:42.554 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.36316.log.gz 2024-03-30T13:28:42.554 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61625.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.61625.log.gzgzip 2024-03-30T13:28:42.554 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40850.log 2024-03-30T13:28:42.554 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31234.log 2024-03-30T13:28:42.554 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43014.log: /var/log/ceph/ceph-client.admin.40850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48130.log 2024-03-30T13:28:42.554 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.43014.log.gz 2024-03-30T13:28:42.555 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.40850.log.gz 2024-03-30T13:28:42.555 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31234.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.31234.log.gz 2024-03-30T13:28:42.555 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53119.log 2024-03-30T13:28:42.555 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61963.log 2024-03-30T13:28:42.555 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48130.log: /var/log/ceph/ceph-client.admin.53119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34755.log 2024-03-30T13:28:42.555 INFO:teuthology.orchestra.run.smithi067.stderr: 47.7% -- replaced with /var/log/ceph/ceph-client.admin.48130.log.gz 2024-03-30T13:28:42.555 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.53119.log.gz 2024-03-30T13:28:42.555 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61963.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.61963.log.gz 2024-03-30T13:28:42.555 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39626.log 2024-03-30T13:28:42.555 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61937.log 2024-03-30T13:28:42.555 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34755.log: /var/log/ceph/ceph-client.admin.39626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38791.log 2024-03-30T13:28:42.555 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.34755.log.gz 2024-03-30T13:28:42.555 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.39626.log.gz/var/log/ceph/ceph-client.admin.61937.log: 2024-03-30T13:28:42.555 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43873.log 2024-03-30T13:28:42.555 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.61937.log.gz 2024-03-30T13:28:42.556 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38791.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.38791.log.gz 2024-03-30T13:28:42.556 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46478.log 2024-03-30T13:28:42.556 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48495.log 2024-03-30T13:28:42.556 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.43873.log.gz 2024-03-30T13:28:42.556 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53406.log 2024-03-30T13:28:42.556 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.46478.log.gz 2024-03-30T13:28:42.556 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48495.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.48495.log.gz 2024-03-30T13:28:42.556 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37070.log 2024-03-30T13:28:42.556 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46556.log 2024-03-30T13:28:42.556 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.53406.log.gz 2024-03-30T13:28:42.556 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48885.log 2024-03-30T13:28:42.556 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.37070.log.gz 2024-03-30T13:28:42.556 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46556.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.46556.log.gz 2024-03-30T13:28:42.556 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33113.log 2024-03-30T13:28:42.556 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32515.log 2024-03-30T13:28:42.556 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48885.log: gzip/var/log/ceph/ceph-client.admin.33113.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.49483.log 2024-03-30T13:28:42.557 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% 47.2% -- replaced with /var/log/ceph/ceph-client.admin.48885.log.gz 2024-03-30T13:28:42.557 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33113.log.gz 2024-03-30T13:28:42.557 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32515.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.32515.log.gz 2024-03-30T13:28:42.557 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32071.log 2024-03-30T13:28:42.557 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38219.log 2024-03-30T13:28:42.557 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49483.log.gz 2024-03-30T13:28:42.557 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36524.log 2024-03-30T13:28:42.557 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.32071.log.gz 2024-03-30T13:28:42.557 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41762.log 2024-03-30T13:28:42.557 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38219.log: /var/log/ceph/ceph-client.admin.36524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53926.log 2024-03-30T13:28:42.557 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.38219.log.gz 2024-03-30T13:28:42.557 INFO:teuthology.orchestra.run.smithi067.stderr: 47.7% -- replaced with /var/log/ceph/ceph-client.admin.36524.log.gz/var/log/ceph/ceph-client.admin.41762.log: 2024-03-30T13:28:42.557 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.41762.log.gz 2024-03-30T13:28:42.557 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41840.log 2024-03-30T13:28:42.557 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34963.log 2024-03-30T13:28:42.557 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53926.log: /var/log/ceph/ceph-client.admin.41840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50501.log 2024-03-30T13:28:42.558 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.53926.log.gz 2024-03-30T13:28:42.558 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.41840.log.gz 2024-03-30T13:28:42.558 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34963.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.34963.log.gz 2024-03-30T13:28:42.558 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63709.log 2024-03-30T13:28:42.558 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32983.log 2024-03-30T13:28:42.558 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50501.log: /var/log/ceph/ceph-client.admin.63709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46842.log 2024-03-30T13:28:42.558 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.50501.log.gz 2024-03-30T13:28:42.558 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.63709.log.gz/var/log/ceph/ceph-client.admin.32983.log: 2024-03-30T13:28:42.558 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.32983.log.gz 2024-03-30T13:28:42.558 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35509.log 2024-03-30T13:28:42.558 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45906.log 2024-03-30T13:28:42.558 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46842.log: /var/log/ceph/ceph-client.admin.35509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39051.log 2024-03-30T13:28:42.558 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.46842.log.gz 2024-03-30T13:28:42.558 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.35509.log.gz 2024-03-30T13:28:42.559 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45906.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.45906.log.gz 2024-03-30T13:28:42.559 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65376.log 2024-03-30T13:28:42.559 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60476.log 2024-03-30T13:28:42.559 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39051.log: /var/log/ceph/ceph-client.admin.65376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36368.log 2024-03-30T13:28:42.559 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.39051.log.gz 2024-03-30T13:28:42.559 INFO:teuthology.orchestra.run.smithi067.stderr: 44.4%/var/log/ceph/ceph-client.admin.60476.log: -- replaced with /var/log/ceph/ceph-client.admin.65376.log.gz 2024-03-30T13:28:42.559 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.60476.log.gz 2024-03-30T13:28:42.559 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55098.log 2024-03-30T13:28:42.559 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52414.log 2024-03-30T13:28:42.559 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36368.log: /var/log/ceph/ceph-client.admin.55098.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.36368.log.gz 2024-03-30T13:28:42.559 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37567.log 2024-03-30T13:28:42.559 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.55098.log.gz 2024-03-30T13:28:42.559 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55306.log 2024-03-30T13:28:42.559 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.52414.log.gz 2024-03-30T13:28:42.559 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30819.log 2024-03-30T13:28:42.559 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37567.log: /var/log/ceph/ceph-client.admin.55306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37177.log 2024-03-30T13:28:42.560 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% 46.7% -- replaced with /var/log/ceph/ceph-client.admin.37567.log.gz 2024-03-30T13:28:42.560 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55306.log.gz 2024-03-30T13:28:42.560 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30819.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.30819.log.gz 2024-03-30T13:28:42.560 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54890.log 2024-03-30T13:28:42.560 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50371.log 2024-03-30T13:28:42.560 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37177.log: /var/log/ceph/ceph-client.admin.54890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51099.log 2024-03-30T13:28:42.560 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.37177.log.gz 47.7% -- replaced with /var/log/ceph/ceph-client.admin.54890.log.gz 2024-03-30T13:28:42.560 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.560 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50371.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.50371.log.gz 2024-03-30T13:28:42.560 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-30T13:28:42.560 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43561.log 2024-03-30T13:28:42.560 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.51099.log.gz 2024-03-30T13:28:42.560 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64886.log 2024-03-30T13:28:42.560 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52203.log 2024-03-30T13:28:42.560 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43561.log: /var/log/ceph/ceph-client.admin.64886.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43561.log.gz 2024-03-30T13:28:42.561 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64886.log.gz 2024-03-30T13:28:42.561 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54708.log 2024-03-30T13:28:42.561 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52203.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.52203.log.gz 2024-03-30T13:28:42.561 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47792.log 2024-03-30T13:28:42.561 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36134.log 2024-03-30T13:28:42.561 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54708.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54708.log.gz 2024-03-30T13:28:42.561 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47792.log: 46.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32229.log 2024-03-30T13:28:42.561 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47792.log.gz 2024-03-30T13:28:42.561 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36134.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.36134.log.gz 2024-03-30T13:28:42.561 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39496.log 2024-03-30T13:28:42.561 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58488.log 2024-03-30T13:28:42.561 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32229.log: /var/log/ceph/ceph-client.admin.39496.log: 46.3% 46.7% -- replaced with /var/log/ceph/ceph-client.admin.39496.log.gz 2024-03-30T13:28:42.561 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32229.log.gz 2024-03-30T13:28:42.561 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40746.log 2024-03-30T13:28:42.561 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39288.log 2024-03-30T13:28:42.561 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.58488.log.gz 2024-03-30T13:28:42.562 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40746.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.40746.log.gz 2024-03-30T13:28:42.562 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32201.log 2024-03-30T13:28:42.562 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49249.log 2024-03-30T13:28:42.562 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.39288.log.gz 2024-03-30T13:28:42.562 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32201.log: 46.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52596.log 2024-03-30T13:28:42.562 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32201.log.gz 2024-03-30T13:28:42.562 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49249.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49249.log.gz 2024-03-30T13:28:42.562 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41502.log 2024-03-30T13:28:42.562 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33581.log 2024-03-30T13:28:42.562 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.52596.log.gz 2024-03-30T13:28:42.562 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41502.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.41502.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50553.log 2024-03-30T13:28:42.562 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.562 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33581.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.33581.log.gz 2024-03-30T13:28:42.562 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49877.log 2024-03-30T13:28:42.562 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40772.log 2024-03-30T13:28:42.563 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.50553.log.gz 2024-03-30T13:28:42.563 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49877.log: 46.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41320.log 2024-03-30T13:28:42.563 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49877.log.gz 2024-03-30T13:28:42.563 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40772.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.40772.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53249.log 2024-03-30T13:28:42.563 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.563 INFO:teuthology.orchestra.run.smithi067.stderr: 92.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-30T13:28:42.563 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41320.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.41320.log.gz 2024-03-30T13:28:42.563 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52336.log 2024-03-30T13:28:42.563 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31806.log 2024-03-30T13:28:42.563 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53249.log: /var/log/ceph/ceph-client.admin.52336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63631.log 2024-03-30T13:28:42.563 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% 46.3% -- replaced with /var/log/ceph/ceph-client.admin.53249.log.gz 2024-03-30T13:28:42.563 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52336.log.gz 2024-03-30T13:28:42.563 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31806.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.31806.log.gz 2024-03-30T13:28:42.563 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50189.log 2024-03-30T13:28:42.563 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60892.log 2024-03-30T13:28:42.563 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63631.log: /var/log/ceph/ceph-client.admin.50189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61989.log 2024-03-30T13:28:42.564 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.63631.log.gz 2024-03-30T13:28:42.564 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.50189.log.gz 2024-03-30T13:28:42.564 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60892.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.60892.log.gz 2024-03-30T13:28:42.564 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64912.log 2024-03-30T13:28:42.564 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31338.log 2024-03-30T13:28:42.564 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61989.log: /var/log/ceph/ceph-client.admin.64912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49119.log 2024-03-30T13:28:42.564 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.61989.log.gz 2024-03-30T13:28:42.564 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64912.log.gz/var/log/ceph/ceph-client.admin.31338.log: 2024-03-30T13:28:42.564 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.31338.log.gz 2024-03-30T13:28:42.564 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58800.log 2024-03-30T13:28:42.564 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50137.log 2024-03-30T13:28:42.564 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49119.log: /var/log/ceph/ceph-client.admin.58800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49197.log 2024-03-30T13:28:42.564 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49119.log.gz 2024-03-30T13:28:42.564 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.58800.log.gz 2024-03-30T13:28:42.565 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50137.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.50137.log.gz 2024-03-30T13:28:42.565 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48104.log 2024-03-30T13:28:42.565 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54525.log 2024-03-30T13:28:42.565 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49197.log: /var/log/ceph/ceph-client.admin.48104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45646.log 2024-03-30T13:28:42.565 INFO:teuthology.orchestra.run.smithi067.stderr: 47.7% -- replaced with /var/log/ceph/ceph-client.admin.48104.log.gz 2024-03-30T13:28:42.565 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49197.log.gz 2024-03-30T13:28:42.565 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54525.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.54525.log.gz 2024-03-30T13:28:42.565 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37229.log 2024-03-30T13:28:42.565 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34443.log 2024-03-30T13:28:42.565 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.45646.log.gz 2024-03-30T13:28:42.565 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46452.log 2024-03-30T13:28:42.565 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.37229.log.gz 2024-03-30T13:28:42.565 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38037.log 2024-03-30T13:28:42.565 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34443.log: /var/log/ceph/ceph-client.admin.46452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58904.log 2024-03-30T13:28:42.565 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.34443.log.gz 2024-03-30T13:28:42.565 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.46452.log.gz 2024-03-30T13:28:42.566 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46712.log 2024-03-30T13:28:42.566 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.38037.log.gz 2024-03-30T13:28:42.566 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58904.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.58904.log.gz 2024-03-30T13:28:42.566 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48026.log 2024-03-30T13:28:42.566 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38947.log 2024-03-30T13:28:42.566 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46712.log: /var/log/ceph/ceph-client.admin.48026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61573.log 2024-03-30T13:28:42.566 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.46712.log.gz 2024-03-30T13:28:42.566 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.48026.log.gz 2024-03-30T13:28:42.566 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38947.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.38947.log.gz 2024-03-30T13:28:42.566 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52073.log 2024-03-30T13:28:42.566 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56350.log 2024-03-30T13:28:42.566 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61573.log: /var/log/ceph/ceph-client.admin.52073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38167.log 2024-03-30T13:28:42.566 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.61573.log.gz 2024-03-30T13:28:42.566 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.52073.log.gz/var/log/ceph/ceph-client.admin.56350.log: 2024-03-30T13:28:42.566 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.56350.log.gz 2024-03-30T13:28:42.567 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36472.log 2024-03-30T13:28:42.567 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60112.log 2024-03-30T13:28:42.567 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38167.log: /var/log/ceph/ceph-client.admin.36472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39914.log 2024-03-30T13:28:42.567 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.38167.log.gz 2024-03-30T13:28:42.567 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.36472.log.gz 2024-03-30T13:28:42.567 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60112.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.60112.log.gz 2024-03-30T13:28:42.567 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56454.log 2024-03-30T13:28:42.567 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52832.log 2024-03-30T13:28:42.567 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39914.log: /var/log/ceph/ceph-client.admin.56454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46816.log 2024-03-30T13:28:42.567 INFO:teuthology.orchestra.run.smithi067.stderr: 47.7% -- replaced with /var/log/ceph/ceph-client.admin.39914.log.gz 2024-03-30T13:28:42.567 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.56454.log.gz/var/log/ceph/ceph-client.admin.52832.log: 2024-03-30T13:28:42.567 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.52832.log.gz 2024-03-30T13:28:42.567 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60945.log 2024-03-30T13:28:42.567 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43379.log 2024-03-30T13:28:42.567 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46816.log: /var/log/ceph/ceph-client.admin.60945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33035.log 2024-03-30T13:28:42.568 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.46816.log.gz 2024-03-30T13:28:42.568 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.60945.log.gz/var/log/ceph/ceph-client.admin.43379.log: 2024-03-30T13:28:42.568 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43379.log.gz 2024-03-30T13:28:42.568 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37044.log 2024-03-30T13:28:42.568 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46790.log 2024-03-30T13:28:42.568 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33035.log: /var/log/ceph/ceph-client.admin.37044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40408.log 2024-03-30T13:28:42.568 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.33035.log.gz 2024-03-30T13:28:42.568 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.37044.log.gz/var/log/ceph/ceph-client.admin.46790.log: 2024-03-30T13:28:42.568 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.46790.log.gz 2024-03-30T13:28:42.568 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50527.log 2024-03-30T13:28:42.568 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65718.log 2024-03-30T13:28:42.568 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40408.log: /var/log/ceph/ceph-client.admin.50527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46374.log 2024-03-30T13:28:42.568 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% 46.7% -- replaced with /var/log/ceph/ceph-client.admin.40408.log.gz 2024-03-30T13:28:42.568 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50527.log.gz 2024-03-30T13:28:42.568 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65718.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.65718.log.gz 2024-03-30T13:28:42.568 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35093.log 2024-03-30T13:28:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54551.log 2024-03-30T13:28:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46374.log: /var/log/ceph/ceph-client.admin.35093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37385.log 2024-03-30T13:28:42.569 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.46374.log.gz 2024-03-30T13:28:42.569 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.35093.log.gz 2024-03-30T13:28:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54551.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54551.log.gz 2024-03-30T13:28:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52047.log 2024-03-30T13:28:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50267.log 2024-03-30T13:28:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37385.log: /var/log/ceph/ceph-client.admin.52047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54864.log 2024-03-30T13:28:42.569 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.37385.log.gz 2024-03-30T13:28:42.569 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.52047.log.gz 2024-03-30T13:28:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50267.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.50267.log.gz 2024-03-30T13:28:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52963.log 2024-03-30T13:28:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40174.log 2024-03-30T13:28:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54864.log: /var/log/ceph/ceph-client.admin.52963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37593.log 2024-03-30T13:28:42.569 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54864.log.gz 2024-03-30T13:28:42.570 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52963.log.gz 2024-03-30T13:28:42.570 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40174.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.40174.log.gz 2024-03-30T13:28:42.570 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57130.log 2024-03-30T13:28:42.570 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54161.log 2024-03-30T13:28:42.570 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37593.log: /var/log/ceph/ceph-client.admin.57130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62327.log 2024-03-30T13:28:42.570 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.37593.log.gz 2024-03-30T13:28:42.570 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2%/var/log/ceph/ceph-client.admin.54161.log: -- replaced with /var/log/ceph/ceph-client.admin.57130.log.gz 2024-03-30T13:28:42.570 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54161.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43040.log 2024-03-30T13:28:42.570 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.570 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54265.log 2024-03-30T13:28:42.570 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62327.log: /var/log/ceph/ceph-client.admin.43040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40642.log 2024-03-30T13:28:42.570 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43040.log.gz 2024-03-30T13:28:42.570 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62327.log.gz 2024-03-30T13:28:42.570 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54265.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54265.log.gz 2024-03-30T13:28:42.570 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43743.log 2024-03-30T13:28:42.571 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51021.log 2024-03-30T13:28:42.571 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40642.log: /var/log/ceph/ceph-client.admin.43743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52989.log 2024-03-30T13:28:42.571 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.40642.log.gz 2024-03-30T13:28:42.571 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7%/var/log/ceph/ceph-client.admin.51021.log: -- replaced with /var/log/ceph/ceph-client.admin.43743.log.gz 2024-03-30T13:28:42.571 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.51021.log.gz 2024-03-30T13:28:42.571 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64808.log 2024-03-30T13:28:42.571 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45255.log 2024-03-30T13:28:42.571 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52989.log: /var/log/ceph/ceph-client.admin.64808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38271.log 2024-03-30T13:28:42.571 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.52989.log.gz 2024-03-30T13:28:42.571 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64808.log.gz/var/log/ceph/ceph-client.admin.45255.log: 2024-03-30T13:28:42.571 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.45255.log.gz 2024-03-30T13:28:42.571 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41110.log 2024-03-30T13:28:42.571 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63553.log 2024-03-30T13:28:42.571 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38271.log: /var/log/ceph/ceph-client.admin.41110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57416.log 2024-03-30T13:28:42.571 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.38271.log.gz 2024-03-30T13:28:42.572 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.41110.log.gz 2024-03-30T13:28:42.572 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63553.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.63553.log.gz 2024-03-30T13:28:42.572 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49743.log 2024-03-30T13:28:42.572 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61205.log 2024-03-30T13:28:42.572 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57416.log: gzip/var/log/ceph/ceph-client.admin.49743.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30602.log 2024-03-30T13:28:42.572 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49743.log.gz 46.7% -- replaced with /var/log/ceph/ceph-client.admin.57416.log.gz 2024-03-30T13:28:42.572 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.572 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61205.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.61205.log.gz 2024-03-30T13:28:42.572 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39470.log 2024-03-30T13:28:42.572 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43431.log 2024-03-30T13:28:42.572 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56220.log 2024-03-30T13:28:42.572 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39470.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.39470.log.gz 2024-03-30T13:28:42.572 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.30602.log.gz 2024-03-30T13:28:42.572 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43431.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.43431.log.gz 2024-03-30T13:28:42.572 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64229.log 2024-03-30T13:28:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32879.log 2024-03-30T13:28:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56220.log: /var/log/ceph/ceph-client.admin.64229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38479.log 2024-03-30T13:28:42.573 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.56220.log.gz 2024-03-30T13:28:42.573 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64229.log.gz 2024-03-30T13:28:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32879.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.32879.log.gz 2024-03-30T13:28:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40226.log 2024-03-30T13:28:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39077.log 2024-03-30T13:28:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35796.log 2024-03-30T13:28:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40226.log: 46.7% 46.7% -- replaced with /var/log/ceph/ceph-client.admin.38479.log.gz 2024-03-30T13:28:42.573 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40226.log.gz 2024-03-30T13:28:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39077.log: 47.7% -- replaced with /var/log/ceph/ceph-client.admin.39077.log.gz 2024-03-30T13:28:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56558.log 2024-03-30T13:28:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40980.log 2024-03-30T13:28:42.573 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.35796.log.gz 2024-03-30T13:28:42.573 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31967.log 2024-03-30T13:28:42.573 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.56558.log.gz 2024-03-30T13:28:42.574 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31915.log 2024-03-30T13:28:42.574 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40980.log: /var/log/ceph/ceph-client.admin.31967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49327.log 2024-03-30T13:28:42.574 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.31967.log.gz 2024-03-30T13:28:42.574 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.40980.log.gz 2024-03-30T13:28:42.574 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31915.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.31915.log.gz 2024-03-30T13:28:42.574 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41866.log 2024-03-30T13:28:42.574 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49587.log 2024-03-30T13:28:42.574 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49327.log.gz 2024-03-30T13:28:42.574 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37151.log 2024-03-30T13:28:42.574 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.41866.log.gz 2024-03-30T13:28:42.574 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49587.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49587.log.gz 2024-03-30T13:28:42.574 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-30T13:28:42.574 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30816.log 2024-03-30T13:28:42.574 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37151.log: /var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38713.log 2024-03-30T13:28:42.574 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.37151.log.gz 2024-03-30T13:28:42.575 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30816.log: 47.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54395.log 2024-03-30T13:28:42.575 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30816.log.gz 2024-03-30T13:28:42.575 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38713.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.38713.log.gz 2024-03-30T13:28:42.575 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61599.log 2024-03-30T13:28:42.575 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50631.log 2024-03-30T13:28:42.575 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54395.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54395.log.gz 2024-03-30T13:28:42.575 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61599.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.61599.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32645.log 2024-03-30T13:28:42.575 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.575 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50631.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.50631.log.gz 2024-03-30T13:28:42.575 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61313.log 2024-03-30T13:28:42.575 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48703.log 2024-03-30T13:28:42.575 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32645.log: /var/log/ceph/ceph-client.admin.61313.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.32645.log.gz 2024-03-30T13:28:42.575 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.61313.log.gz 2024-03-30T13:28:42.575 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48364.log 2024-03-30T13:28:42.575 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46088.log 2024-03-30T13:28:42.575 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48703.log: /var/log/ceph/ceph-client.admin.48364.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.48703.log.gz 2024-03-30T13:28:42.576 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.48364.log.gz 2024-03-30T13:28:42.576 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34573.log 2024-03-30T13:28:42.576 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36056.log 2024-03-30T13:28:42.576 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46088.log: /var/log/ceph/ceph-client.admin.34573.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.46088.log.gz 2024-03-30T13:28:42.576 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.34573.log.gz 2024-03-30T13:28:42.576 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32437.log 2024-03-30T13:28:42.576 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41268.log 2024-03-30T13:28:42.576 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36056.log: /var/log/ceph/ceph-client.admin.32437.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.36056.log.gz 2024-03-30T13:28:42.576 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.32437.log.gz 2024-03-30T13:28:42.576 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52726.log 2024-03-30T13:28:42.576 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58436.log 2024-03-30T13:28:42.576 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41268.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.41268.log.gz 2024-03-30T13:28:42.576 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52726.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.52726.log.gz 2024-03-30T13:28:42.576 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63735.log 2024-03-30T13:28:42.576 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58436.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.58436.log.gz 2024-03-30T13:28:42.577 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44003.log 2024-03-30T13:28:42.577 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31025.log 2024-03-30T13:28:42.577 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63735.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.63735.log.gz 2024-03-30T13:28:42.577 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.44003.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.44003.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55124.log 2024-03-30T13:28:42.577 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.577 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31025.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.31025.log.gz 2024-03-30T13:28:42.577 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54421.log 2024-03-30T13:28:42.577 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57104.log 2024-03-30T13:28:42.577 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55124.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.55124.log.gz 2024-03-30T13:28:42.577 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54421.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54421.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57364.log 2024-03-30T13:28:42.577 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.577 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57104.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.57104.log.gz 2024-03-30T13:28:42.577 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53614.log 2024-03-30T13:28:42.577 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37881.log 2024-03-30T13:28:42.577 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57364.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.57364.log.gz 2024-03-30T13:28:42.578 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53614.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.53614.log.gzgzip 2024-03-30T13:28:42.578 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52674.log 2024-03-30T13:28:42.578 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37881.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.37881.log.gz 2024-03-30T13:28:42.578 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35145.log 2024-03-30T13:28:42.578 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35483.log 2024-03-30T13:28:42.578 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52674.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.52674.log.gz 2024-03-30T13:28:42.578 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35145.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.35145.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38895.log 2024-03-30T13:28:42.578 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.578 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35483.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.35483.log.gz 2024-03-30T13:28:42.578 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50007.log 2024-03-30T13:28:42.578 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37489.log 2024-03-30T13:28:42.578 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38895.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.38895.log.gz 2024-03-30T13:28:42.578 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50007.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.50007.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50605.log 2024-03-30T13:28:42.578 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.578 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37489.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.37489.log.gz 2024-03-30T13:28:42.579 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41632.log 2024-03-30T13:28:42.579 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31640.log 2024-03-30T13:28:42.579 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50605.log: /var/log/ceph/ceph-client.admin.41632.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.50605.log.gz 2024-03-30T13:28:42.579 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.41632.log.gz 2024-03-30T13:28:42.579 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37281.log 2024-03-30T13:28:42.579 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57026.log 2024-03-30T13:28:42.579 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31640.log: /var/log/ceph/ceph-client.admin.37281.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.31640.log.gz 2024-03-30T13:28:42.579 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.37281.log.gz 2024-03-30T13:28:42.579 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39574.log 2024-03-30T13:28:42.579 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37203.log 2024-03-30T13:28:42.579 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57026.log: /var/log/ceph/ceph-client.admin.39574.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.57026.log.gz 2024-03-30T13:28:42.579 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.39574.log.gz 2024-03-30T13:28:42.579 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43222.log 2024-03-30T13:28:42.579 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45515.log 2024-03-30T13:28:42.579 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37203.log: /var/log/ceph/ceph-client.admin.43222.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.37203.log.gz 2024-03-30T13:28:42.580 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43222.log.gz 2024-03-30T13:28:42.580 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63761.log 2024-03-30T13:28:42.580 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58540.log 2024-03-30T13:28:42.580 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45515.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.45515.log.gz 2024-03-30T13:28:42.580 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63761.log: 46.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50891.log 2024-03-30T13:28:42.580 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63761.log.gz 2024-03-30T13:28:42.580 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58540.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.58540.log.gz 2024-03-30T13:28:42.580 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50943.log 2024-03-30T13:28:42.580 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56376.log 2024-03-30T13:28:42.580 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50891.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.50891.log.gz 2024-03-30T13:28:42.580 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50943.log: 46.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55280.log 2024-03-30T13:28:42.580 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50943.log.gz 2024-03-30T13:28:42.580 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56376.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.56376.log.gz 2024-03-30T13:28:42.580 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43613.log 2024-03-30T13:28:42.580 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35900.log 2024-03-30T13:28:42.580 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55280.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.55280.log.gz 2024-03-30T13:28:42.581 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43613.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43613.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66082.log 2024-03-30T13:28:42.581 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.581 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35900.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.35900.log.gz 2024-03-30T13:28:42.581 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25025.log 2024-03-30T13:28:42.581 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42988.log 2024-03-30T13:28:42.581 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66082.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.66082.log.gz 2024-03-30T13:28:42.581 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.25025.log: 45.8% -- replaced with /var/log/ceph/ceph-client.admin.25025.log.gz 2024-03-30T13:28:42.581 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36992.log 2024-03-30T13:28:42.581 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42988.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.42988.log.gz 2024-03-30T13:28:42.581 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40512.log 2024-03-30T13:28:42.581 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41606.log 2024-03-30T13:28:42.581 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40512.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.36992.log.gz 2024-03-30T13:28:42.581 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.40512.log.gz 2024-03-30T13:28:42.581 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45958.log 2024-03-30T13:28:42.581 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41606.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.41606.log.gz 2024-03-30T13:28:42.581 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39652.log 2024-03-30T13:28:42.582 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50293.log 2024-03-30T13:28:42.582 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45958.log: /var/log/ceph/ceph-client.admin.39652.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.39652.log.gz 2024-03-30T13:28:42.582 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.45958.log.gz 2024-03-30T13:28:42.582 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54994.log 2024-03-30T13:28:42.582 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37907.log 2024-03-30T13:28:42.582 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.50293.log.gz 2024-03-30T13:28:42.582 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54291.log 2024-03-30T13:28:42.582 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54994.log: /var/log/ceph/ceph-client.admin.37907.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.37907.log.gz 2024-03-30T13:28:42.582 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54994.log.gz 2024-03-30T13:28:42.582 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61495.log 2024-03-30T13:28:42.582 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54291.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54291.log.gz 2024-03-30T13:28:42.582 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66186.log 2024-03-30T13:28:42.582 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52806.log 2024-03-30T13:28:42.582 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.61495.log.gz 2024-03-30T13:28:42.582 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66186.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.66186.log.gz 2024-03-30T13:28:42.583 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45307.log 2024-03-30T13:28:42.583 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37855.log 2024-03-30T13:28:42.583 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52806.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.52806.log.gz 2024-03-30T13:28:42.636 INFO:teuthology.orchestra.run.smithi082.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-30T13:28:42.638 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45307.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.45307.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45620.log 2024-03-30T13:28:42.638 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.638 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50735.log 2024-03-30T13:28:42.638 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.37855.log.gz 2024-03-30T13:28:42.638 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45620.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.45620.log.gz 2024-03-30T13:28:42.638 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38557.log 2024-03-30T13:28:42.638 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57182.log 2024-03-30T13:28:42.638 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.50735.log.gz 2024-03-30T13:28:42.638 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38557.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.38557.log.gz 2024-03-30T13:28:42.639 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30506.log 2024-03-30T13:28:42.639 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47844.log 2024-03-30T13:28:42.639 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57182.log: /var/log/ceph/ceph-client.admin.30506.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.57182.log.gz 2024-03-30T13:28:42.639 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.30506.log.gz 2024-03-30T13:28:42.639 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41476.log 2024-03-30T13:28:42.639 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47844.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.47844.log.gz 2024-03-30T13:28:42.639 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46400.log 2024-03-30T13:28:42.639 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60632.log 2024-03-30T13:28:42.639 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41476.log: /var/log/ceph/ceph-client.admin.46400.log: 46.3% 47.2% -- replaced with /var/log/ceph/ceph-client.admin.46400.log.gz 2024-03-30T13:28:42.639 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41476.log.gz 2024-03-30T13:28:42.639 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59930.log 2024-03-30T13:28:42.639 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32853.log 2024-03-30T13:28:42.639 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.60632.log.gz 2024-03-30T13:28:42.639 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36498.log 2024-03-30T13:28:42.639 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.59930.log: /var/log/ceph/ceph-client.admin.32853.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.59930.log.gz 46.7% -- replaced with /var/log/ceph/ceph-client.admin.32853.log.gz 2024-03-30T13:28:42.640 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49535.log 2024-03-30T13:28:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49457.log 2024-03-30T13:28:42.640 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.36498.log.gz 2024-03-30T13:28:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46296.log 2024-03-30T13:28:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49535.log: /var/log/ceph/ceph-client.admin.49457.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49535.log.gz 2024-03-30T13:28:42.640 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.49457.log.gz 2024-03-30T13:28:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49145.log 2024-03-30T13:28:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38063.log 2024-03-30T13:28:42.640 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.46296.log.gz 2024-03-30T13:28:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49145.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49145.log.gz 2024-03-30T13:28:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39155.log 2024-03-30T13:28:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39522.log 2024-03-30T13:28:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38063.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.38063.log.gz 2024-03-30T13:28:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39155.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.39155.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25200.log 2024-03-30T13:28:42.640 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39522.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.39522.log.gz 2024-03-30T13:28:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39704.log 2024-03-30T13:28:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.25200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31594.log 2024-03-30T13:28:42.641 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.25200.log.gz 2024-03-30T13:28:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39704.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.39704.log.gz 2024-03-30T13:28:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38635.log 2024-03-30T13:28:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38973.log 2024-03-30T13:28:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31594.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.31594.log.gz 2024-03-30T13:28:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38635.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.38635.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42962.log 2024-03-30T13:28:42.641 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38973.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.38973.log.gz 2024-03-30T13:28:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39600.log 2024-03-30T13:28:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57000.log 2024-03-30T13:28:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42962.log: /var/log/ceph/ceph-client.admin.39600.log: 47.7% -- replaced with /var/log/ceph/ceph-client.admin.42962.log.gz 2024-03-30T13:28:42.641 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.39600.log.gz 2024-03-30T13:28:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47948.log 2024-03-30T13:28:42.642 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57156.log 2024-03-30T13:28:42.642 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57000.log: /var/log/ceph/ceph-client.admin.47948.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.57000.log.gz 2024-03-30T13:28:42.642 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.47948.log.gz 2024-03-30T13:28:42.642 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-30T13:28:42.642 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61469.log 2024-03-30T13:28:42.642 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57156.log: /var/log/ceph/ceph.audit.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.57156.log.gz 2024-03-30T13:28:42.642 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46608.log 2024-03-30T13:28:42.642 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61469.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.61469.log.gz 2024-03-30T13:28:42.642 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33139.log 2024-03-30T13:28:42.642 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46608.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.46608.log.gz 2024-03-30T13:28:42.642 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56428.log 2024-03-30T13:28:42.642 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33139.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.33139.log.gz 2024-03-30T13:28:42.642 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54577.log 2024-03-30T13:28:42.642 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56428.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.56428.log.gz 2024-03-30T13:28:42.642 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33165.log 2024-03-30T13:28:42.643 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54577.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54577.log.gz 2024-03-30T13:28:42.643 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40876.log 2024-03-30T13:28:42.643 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33165.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.33165.log.gz 2024-03-30T13:28:42.643 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38739.log 2024-03-30T13:28:42.643 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40876.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.40876.log.gz 2024-03-30T13:28:42.643 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31260.log 2024-03-30T13:28:42.643 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38739.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.38739.log.gz 2024-03-30T13:28:42.643 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53067.log 2024-03-30T13:28:42.643 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31260.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.31260.log.gz 2024-03-30T13:28:42.643 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32255.log 2024-03-30T13:28:42.643 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53067.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.53067.log.gz 2024-03-30T13:28:42.643 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52362.log 2024-03-30T13:28:42.643 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32255.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.32255.log.gz 2024-03-30T13:28:42.643 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54031.log 2024-03-30T13:28:42.643 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52362.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.52362.log.gz 2024-03-30T13:28:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65381.log 2024-03-30T13:28:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54031.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54031.log.gz 2024-03-30T13:28:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53380.log 2024-03-30T13:28:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65381.log: 44.4% -- replaced with /var/log/ceph/ceph-client.admin.65381.log.gz 2024-03-30T13:28:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63813.log 2024-03-30T13:28:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53380.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.53380.log.gz 2024-03-30T13:28:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38349.log 2024-03-30T13:28:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63813.log: 94.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-30T13:28:42.644 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.63813.log.gz 2024-03-30T13:28:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66134.log 2024-03-30T13:28:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65382.log 2024-03-30T13:28:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38349.log: /var/log/ceph/ceph-client.admin.66134.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.38349.log.gz 2024-03-30T13:28:42.644 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.66134.log.gz 2024-03-30T13:28:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64021.log 2024-03-30T13:28:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45411.log 2024-03-30T13:28:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65382.log: /var/log/ceph/ceph-client.admin.64021.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.65382.log.gz 2024-03-30T13:28:42.645 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64021.log.gz 2024-03-30T13:28:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56324.log 2024-03-30T13:28:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31364.log 2024-03-30T13:28:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56324.log: /var/log/ceph/ceph-client.admin.45411.log: 46.7% 47.2% -- replaced with /var/log/ceph/ceph-client.admin.45411.log.gz 2024-03-30T13:28:42.645 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56324.log.gz 2024-03-30T13:28:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33321.log 2024-03-30T13:28:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48573.log 2024-03-30T13:28:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31364.log: /var/log/ceph/ceph-client.admin.33321.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.31364.log.gz 2024-03-30T13:28:42.645 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.33321.log.gz 2024-03-30T13:28:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38661.log 2024-03-30T13:28:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56298.log 2024-03-30T13:28:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48573.log: /var/log/ceph/ceph-client.admin.38661.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.48573.log.gz 2024-03-30T13:28:42.645 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.38661.log.gz 2024-03-30T13:28:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33425.log 2024-03-30T13:28:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43951.log 2024-03-30T13:28:42.646 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56298.log: /var/log/ceph/ceph-client.admin.33425.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.56298.log.gz 2024-03-30T13:28:42.646 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.33425.log.gz 2024-03-30T13:28:42.646 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43587.log 2024-03-30T13:28:42.646 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64411.log 2024-03-30T13:28:42.646 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43951.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43951.log.gz 2024-03-30T13:28:42.646 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43587.log: 46.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.43587.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.48156.log 2024-03-30T13:28:42.646 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.646 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64411.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.64411.log.gz 2024-03-30T13:28:42.646 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36082.log 2024-03-30T13:28:42.646 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57078.log 2024-03-30T13:28:42.646 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48156.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.48156.log.gz 2024-03-30T13:28:42.646 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36082.log: 46.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38401.log 2024-03-30T13:28:42.646 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36082.log.gz 2024-03-30T13:28:42.646 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57078.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.57078.log.gz 2024-03-30T13:28:42.646 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62145.log 2024-03-30T13:28:42.647 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63501.log 2024-03-30T13:28:42.647 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38401.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.38401.log.gz 2024-03-30T13:28:42.647 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62145.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.62145.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48729.log 2024-03-30T13:28:42.647 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.647 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63501.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.63501.log.gz 2024-03-30T13:28:42.647 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35353.log 2024-03-30T13:28:42.647 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48052.log 2024-03-30T13:28:42.647 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48729.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.48729.log.gz 2024-03-30T13:28:42.647 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35353.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.35353.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56506.log 2024-03-30T13:28:42.647 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.647 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48052.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.48052.log.gz 2024-03-30T13:28:42.647 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65151.log 2024-03-30T13:28:42.647 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53770.log 2024-03-30T13:28:42.647 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56506.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.56506.log.gz 2024-03-30T13:28:42.647 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65151.log: 47.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30804.log 2024-03-30T13:28:42.648 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.65151.log.gz 2024-03-30T13:28:42.648 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53770.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.53770.log.gz 2024-03-30T13:28:42.648 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43457.log 2024-03-30T13:28:42.648 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37437.log 2024-03-30T13:28:42.648 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30804.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.30804.log.gz 2024-03-30T13:28:42.648 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43457.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43457.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32385.log 2024-03-30T13:28:42.648 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.648 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37437.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.37437.log.gz 2024-03-30T13:28:42.648 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49769.log 2024-03-30T13:28:42.648 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54369.log 2024-03-30T13:28:42.648 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49769.log: /var/log/ceph/ceph-client.admin.32385.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.32385.log.gz 2024-03-30T13:28:42.648 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49769.log.gz 2024-03-30T13:28:42.648 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45177.log 2024-03-30T13:28:42.648 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43144.log 2024-03-30T13:28:42.648 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54369.log: /var/log/ceph/ceph-client.admin.45177.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54369.log.gz 2024-03-30T13:28:42.648 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.45177.log.gz 2024-03-30T13:28:42.649 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40018.log 2024-03-30T13:28:42.649 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52466.log 2024-03-30T13:28:42.649 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43144.log: /var/log/ceph/ceph-client.admin.40018.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43144.log.gz 2024-03-30T13:28:42.649 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.40018.log.gz 2024-03-30T13:28:42.649 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60554.log 2024-03-30T13:28:42.649 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41006.log 2024-03-30T13:28:42.649 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52466.log: /var/log/ceph/ceph-client.admin.60554.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.52466.log.gz 2024-03-30T13:28:42.649 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.60554.log.gz 2024-03-30T13:28:42.649 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32931.log 2024-03-30T13:28:42.649 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53744.log 2024-03-30T13:28:42.649 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41006.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.41006.log.gz 2024-03-30T13:28:42.649 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32931.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.32931.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45854.log 2024-03-30T13:28:42.649 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.649 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53744.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.53744.log.gz 2024-03-30T13:28:42.649 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48963.log 2024-03-30T13:28:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62223.log 2024-03-30T13:28:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45854.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.45854.log.gz 2024-03-30T13:28:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48963.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.48963.log.gz 2024-03-30T13:28:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49981.log 2024-03-30T13:28:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62223.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.62223.log.gz 2024-03-30T13:28:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48521.log 2024-03-30T13:28:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39262.log 2024-03-30T13:28:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49981.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49981.log.gz 2024-03-30T13:28:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48521.log: 47.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.48521.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.49405.log 2024-03-30T13:28:42.650 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39262.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.39262.log.gz 2024-03-30T13:28:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40148.log 2024-03-30T13:28:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32307.log 2024-03-30T13:28:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49405.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49405.log.gz 2024-03-30T13:28:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40148.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.40148.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39210.log 2024-03-30T13:28:42.651 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32307.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.32307.log.gz 2024-03-30T13:28:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37933.log 2024-03-30T13:28:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61261.log 2024-03-30T13:28:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39210.log: /var/log/ceph/ceph-client.admin.37933.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.39210.log.gz 2024-03-30T13:28:42.651 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.37933.log.gz 2024-03-30T13:28:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48937.log 2024-03-30T13:28:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61261.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.61261.log.gz 2024-03-30T13:28:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46218.log 2024-03-30T13:28:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56766.log 2024-03-30T13:28:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48937.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.48937.log.gz 2024-03-30T13:28:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46218.log: 46.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58748.log 2024-03-30T13:28:42.651 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46218.log.gz 2024-03-30T13:28:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56766.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.56766.log.gz 2024-03-30T13:28:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39314.log 2024-03-30T13:28:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32749.log 2024-03-30T13:28:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58748.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.58748.log.gz 2024-03-30T13:28:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39314.log: 46.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.39314.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.36550.log 2024-03-30T13:28:42.652 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32749.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.32749.log.gz 2024-03-30T13:28:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50761.log 2024-03-30T13:28:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35744.log 2024-03-30T13:28:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36550.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.36550.log.gz 2024-03-30T13:28:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50761.log: 46.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36888.log 2024-03-30T13:28:42.652 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50761.log.gz 2024-03-30T13:28:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35744.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.35744.log.gz 2024-03-30T13:28:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39678.log 2024-03-30T13:28:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33399.log 2024-03-30T13:28:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36888.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.36888.log.gz 2024-03-30T13:28:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39678.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.39678.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55046.log 2024-03-30T13:28:42.652 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33399.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.33399.log.gz 2024-03-30T13:28:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66056.log 2024-03-30T13:28:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36004.log 2024-03-30T13:28:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.55046.log: /var/log/ceph/ceph-client.admin.66056.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.66056.log.gz 2024-03-30T13:28:42.653 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.55046.log.gz 2024-03-30T13:28:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35327.log 2024-03-30T13:28:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45047.log 2024-03-30T13:28:42.653 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.36004.log.gz 2024-03-30T13:28:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45125.log 2024-03-30T13:28:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35327.log: /var/log/ceph/ceph-client.admin.45047.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.35327.log.gz 2024-03-30T13:28:42.653 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.45047.log.gz 2024-03-30T13:28:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62379.log 2024-03-30T13:28:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65378.log 2024-03-30T13:28:42.653 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.45125.log.gz 2024-03-30T13:28:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62379.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.62379.log.gz 2024-03-30T13:28:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25129.log 2024-03-30T13:28:42.654 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43769.log 2024-03-30T13:28:42.654 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65378.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.65378.log.gz 2024-03-30T13:28:42.654 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.25129.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.25129.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54916.log 2024-03-30T13:28:42.654 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.654 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43769.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43769.log.gz 2024-03-30T13:28:42.654 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52388.log 2024-03-30T13:28:42.654 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53041.log 2024-03-30T13:28:42.654 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54916.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.54916.log.gz 2024-03-30T13:28:42.654 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52388.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.52388.log.gzgzip 2024-03-30T13:28:42.654 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.62431.log 2024-03-30T13:28:42.654 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53041.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.53041.log.gz 2024-03-30T13:28:42.654 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60762.log 2024-03-30T13:28:42.654 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36680.log 2024-03-30T13:28:42.654 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62431.log: /var/log/ceph/ceph-client.admin.60762.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.62431.log.gz 2024-03-30T13:28:42.654 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.60762.log.gz 2024-03-30T13:28:42.655 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60788.log 2024-03-30T13:28:42.655 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60346.log 2024-03-30T13:28:42.655 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36680.log: /var/log/ceph/ceph-client.admin.60788.log: 44.6% -- replaced with /var/log/ceph/ceph-client.admin.36680.log.gz 2024-03-30T13:28:42.655 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.60788.log.gz 2024-03-30T13:28:42.655 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30817.log 2024-03-30T13:28:42.655 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57260.log 2024-03-30T13:28:42.655 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60346.log: 47.2%/var/log/ceph/ceph-client.admin.30817.log: -- replaced with /var/log/ceph/ceph-client.admin.60346.log.gz 2024-03-30T13:28:42.655 INFO:teuthology.orchestra.run.smithi067.stderr: 44.9% -- replaced with /var/log/ceph/ceph-client.admin.30817.log.gz 2024-03-30T13:28:42.655 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37255.log 2024-03-30T13:28:42.655 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57260.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.57260.log.gz 2024-03-30T13:28:42.655 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47896.log 2024-03-30T13:28:42.655 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52229.log 2024-03-30T13:28:42.655 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37255.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.37255.log.gz 2024-03-30T13:28:42.655 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.47896.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.47896.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41450.log 2024-03-30T13:28:42.655 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52229.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.52229.log.gz 2024-03-30T13:28:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46244.log 2024-03-30T13:28:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43977.log 2024-03-30T13:28:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41450.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.41450.log.gz 2024-03-30T13:28:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46244.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.46244.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45359.log 2024-03-30T13:28:42.656 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43977.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43977.log.gz 2024-03-30T13:28:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40070.log 2024-03-30T13:28:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39782.log 2024-03-30T13:28:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45359.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.45359.log.gz 2024-03-30T13:28:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40070.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.40070.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60840.log 2024-03-30T13:28:42.656 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39782.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.39782.log.gz 2024-03-30T13:28:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38115.log 2024-03-30T13:28:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37775.log 2024-03-30T13:28:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60840.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.60840.log.gz 2024-03-30T13:28:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38115.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.38115.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48547.log 2024-03-30T13:28:42.657 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37775.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.37775.log.gz 2024-03-30T13:28:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57312.log 2024-03-30T13:28:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65362.log 2024-03-30T13:28:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48547.log: 48.1% -- replaced with /var/log/ceph/ceph-client.admin.48547.log.gz 2024-03-30T13:28:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57312.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.57312.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41058.log 2024-03-30T13:28:42.657 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65362.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.65362.log.gz 2024-03-30T13:28:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64652.log 2024-03-30T13:28:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56662.log 2024-03-30T13:28:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41058.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.41058.log.gz 2024-03-30T13:28:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64652.log: 46.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62041.log 2024-03-30T13:28:42.657 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.64652.log.gz 2024-03-30T13:28:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56662.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.56662.log.gz 2024-03-30T13:28:42.658 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54682.log 2024-03-30T13:28:42.658 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54734.log 2024-03-30T13:28:42.658 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62041.log: /var/log/ceph/ceph-client.admin.54682.log: 46.7% 47.2% -- replaced with /var/log/ceph/ceph-client.admin.54682.log.gz 2024-03-30T13:28:42.658 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62041.log.gz 2024-03-30T13:28:42.658 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50709.log 2024-03-30T13:28:42.683 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33503.log 2024-03-30T13:28:42.683 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54734.log.gz 2024-03-30T13:28:42.683 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42858.log 2024-03-30T13:28:42.683 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50709.log: /var/log/ceph/ceph-client.admin.33503.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.50709.log.gz 46.7% -- replaced with /var/log/ceph/ceph-client.admin.33503.log.gz 2024-03-30T13:28:42.683 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.683 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56948.log 2024-03-30T13:28:42.683 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.42858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64704.log 2024-03-30T13:28:42.684 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.42858.log.gz 2024-03-30T13:28:42.684 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48911.log 2024-03-30T13:28:42.684 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56948.log: /var/log/ceph/ceph-client.admin.64704.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.56948.log.gz 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64704.log.gz 2024-03-30T13:28:42.684 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.684 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37671.log 2024-03-30T13:28:42.684 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64860.log 2024-03-30T13:28:42.684 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.48911.log.gz 2024-03-30T13:28:42.684 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54942.log 2024-03-30T13:28:42.684 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37671.log: /var/log/ceph/ceph-client.admin.64860.log: 46.7% 46.7% -- replaced with /var/log/ceph/ceph-client.admin.37671.log.gz -- replaced with /var/log/ceph/ceph-client.admin.64860.log.gz 2024-03-30T13:28:42.684 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.684 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48182.log 2024-03-30T13:28:42.684 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54942.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54942.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58644.log 2024-03-30T13:28:42.684 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.684 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48182.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.48182.log.gz 2024-03-30T13:28:42.684 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61807.log 2024-03-30T13:28:42.685 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48859.log 2024-03-30T13:28:42.685 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58644.log: /var/log/ceph/ceph-client.admin.61807.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.58644.log.gz 2024-03-30T13:28:42.685 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.61807.log.gz 2024-03-30T13:28:42.685 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41136.log 2024-03-30T13:28:42.685 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54760.log 2024-03-30T13:28:42.685 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48859.log: /var/log/ceph/ceph-client.admin.41136.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.48859.log.gz 2024-03-30T13:28:42.685 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.41136.log.gz 2024-03-30T13:28:42.685 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37749.log 2024-03-30T13:28:42.685 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65692.log 2024-03-30T13:28:42.685 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54760.log: /var/log/ceph/ceph-client.admin.37749.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54760.log.gz 2024-03-30T13:28:42.685 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.37749.log.gz 2024-03-30T13:28:42.685 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36394.log 2024-03-30T13:28:42.685 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58696.log 2024-03-30T13:28:42.685 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65692.log: /var/log/ceph/ceph-client.admin.36394.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.65692.log.gz 2024-03-30T13:28:42.685 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.36394.log.gz 2024-03-30T13:28:42.685 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45724.log 2024-03-30T13:28:42.686 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52177.log 2024-03-30T13:28:42.686 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58696.log: /var/log/ceph/ceph-client.admin.45724.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.58696.log.gz 2024-03-30T13:28:42.686 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.45724.log.gz 2024-03-30T13:28:42.686 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31734.log 2024-03-30T13:28:42.686 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36836.log 2024-03-30T13:28:42.686 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52177.log: /var/log/ceph/ceph-client.admin.31734.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.52177.log.gz 2024-03-30T13:28:42.686 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.31734.log.gz 2024-03-30T13:28:42.686 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43483.log 2024-03-30T13:28:42.686 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32827.log 2024-03-30T13:28:42.686 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7%/var/log/ceph/ceph-client.admin.43483.log: -- replaced with /var/log/ceph/ceph-client.admin.36836.log.gz 2024-03-30T13:28:42.686 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43483.log.gz 2024-03-30T13:28:42.686 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45541.log 2024-03-30T13:28:42.686 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32827.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.32827.log.gz 2024-03-30T13:28:42.686 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63839.log 2024-03-30T13:28:42.687 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51047.log 2024-03-30T13:28:42.687 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45541.log: /var/log/ceph/ceph-client.admin.63839.log: 46.3% 46.7% -- replaced with /var/log/ceph/ceph-client.admin.63839.log.gz 2024-03-30T13:28:42.687 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45541.log.gz 2024-03-30T13:28:42.687 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36706.log 2024-03-30T13:28:42.687 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.51047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32281.log 2024-03-30T13:28:42.687 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.51047.log.gz 2024-03-30T13:28:42.687 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36706.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.36706.log.gz 2024-03-30T13:28:42.687 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50839.log 2024-03-30T13:28:42.687 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41242.log 2024-03-30T13:28:42.687 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32281.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.32281.log.gz 2024-03-30T13:28:42.687 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50839.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.50839.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34833.log 2024-03-30T13:28:42.687 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.687 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41242.log: 45.2% -- replaced with /var/log/ceph/ceph-client.admin.41242.log.gz 2024-03-30T13:28:42.687 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37097.log 2024-03-30T13:28:42.687 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50423.log 2024-03-30T13:28:42.688 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34833.log: /var/log/ceph/ceph-client.admin.37097.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.34833.log.gz 2024-03-30T13:28:42.688 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.37097.log.gz 2024-03-30T13:28:42.688 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37359.log 2024-03-30T13:28:42.688 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38141.log 2024-03-30T13:28:42.688 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50423.log: /var/log/ceph/ceph-client.admin.37359.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.50423.log.gz 2024-03-30T13:28:42.688 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.37359.log.gz 2024-03-30T13:28:42.688 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32593.log 2024-03-30T13:28:42.688 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34781.log 2024-03-30T13:28:42.688 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38141.log: /var/log/ceph/ceph-client.admin.32593.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.38141.log.gz 2024-03-30T13:28:42.688 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.32593.log.gz 2024-03-30T13:28:42.688 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65952.log 2024-03-30T13:28:42.688 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49613.log 2024-03-30T13:28:42.688 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34781.log: /var/log/ceph/ceph-client.admin.65952.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.34781.log.gz 2024-03-30T13:28:42.688 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.65952.log.gz 2024-03-30T13:28:42.688 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50449.log 2024-03-30T13:28:42.688 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60710.log 2024-03-30T13:28:42.689 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49613.log: /var/log/ceph/ceph-client.admin.50449.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.49613.log.gz 2024-03-30T13:28:42.689 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.50449.log.gz 2024-03-30T13:28:42.689 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54005.log 2024-03-30T13:28:42.689 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66030.log 2024-03-30T13:28:42.689 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60710.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.60710.log.gz 2024-03-30T13:28:42.689 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54005.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.54005.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57338.log 2024-03-30T13:28:42.689 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.689 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66030.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.66030.log.gz 2024-03-30T13:28:42.689 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52255.log 2024-03-30T13:28:42.689 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32541.log 2024-03-30T13:28:42.689 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57338.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.57338.log.gz 2024-03-30T13:28:42.689 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52255.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.52255.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62483.log 2024-03-30T13:28:42.689 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.689 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32541.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.32541.log.gz 2024-03-30T13:28:42.689 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30802.log 2024-03-30T13:28:42.690 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36914.log 2024-03-30T13:28:42.690 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62483.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.62483.log.gz 2024-03-30T13:28:42.690 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.30802.log: 44.4% -- replaced with /var/log/ceph/ceph-client.admin.30802.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53510.log 2024-03-30T13:28:42.690 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.690 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36914.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.36914.log.gz 2024-03-30T13:28:42.690 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66210.log 2024-03-30T13:28:42.690 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40096.log 2024-03-30T13:28:42.690 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53510.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.53510.log.gz 2024-03-30T13:28:42.690 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66210.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.66210.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41372.log 2024-03-30T13:28:42.690 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.690 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40096.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.40096.log.gz 2024-03-30T13:28:42.690 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35822.log 2024-03-30T13:28:42.690 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65900.log 2024-03-30T13:28:42.690 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41372.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.41372.log.gz 2024-03-30T13:28:42.690 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35822.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.35822.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53666.log 2024-03-30T13:28:42.691 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.691 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65900.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.65900.log.gz 2024-03-30T13:28:42.691 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61153.log 2024-03-30T13:28:42.691 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60424.log 2024-03-30T13:28:42.691 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.53666.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.53666.log.gz 2024-03-30T13:28:42.691 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61153.log: 46.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62119.log 2024-03-30T13:28:42.691 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.61153.log.gz 2024-03-30T13:28:42.691 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60424.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.60424.log.gz 2024-03-30T13:28:42.691 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40382.log 2024-03-30T13:28:42.691 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35171.log 2024-03-30T13:28:42.691 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62119.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.62119.log.gz 2024-03-30T13:28:42.691 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40382.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.40382.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38245.log 2024-03-30T13:28:42.692 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.692 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35171.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.35171.log.gz 2024-03-30T13:28:42.692 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64385.log 2024-03-30T13:28:42.692 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39888.log 2024-03-30T13:28:42.692 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38245.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.38245.log.gz 2024-03-30T13:28:42.692 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64385.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64385.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46894.log 2024-03-30T13:28:42.692 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.692 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39888.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.39888.log.gz 2024-03-30T13:28:42.692 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48677.log 2024-03-30T13:28:42.692 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43092.log 2024-03-30T13:28:42.692 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46894.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.46894.log.gz 2024-03-30T13:28:42.692 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48677.log: gzip 46.7% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.48677.log.gz -- 2024-03-30T13:28:42.692 INFO:teuthology.orchestra.run.smithi067.stderr: /var/log/ceph/ceph-client.admin.46660.log 2024-03-30T13:28:42.692 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60294.log 2024-03-30T13:28:42.692 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.43092.log: /var/log/ceph/ceph-client.admin.46660.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.46660.log.gz 2024-03-30T13:28:42.693 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.43092.log.gz 2024-03-30T13:28:42.693 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35614.log 2024-03-30T13:28:42.693 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43795.log 2024-03-30T13:28:42.693 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.60294.log.gz 2024-03-30T13:28:42.693 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61911.log 2024-03-30T13:28:42.693 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35614.log: /var/log/ceph/ceph-client.admin.43795.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.35614.log.gz 47.2% -- replaced with /var/log/ceph/ceph-client.admin.43795.log.gz 2024-03-30T13:28:42.693 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.693 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56896.log 2024-03-30T13:28:42.693 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41788.log 2024-03-30T13:28:42.693 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.61911.log.gz 2024-03-30T13:28:42.708 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35067.log 2024-03-30T13:28:42.709 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.41788.log: /var/log/ceph/ceph-client.admin.56896.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.41788.log.gz 2024-03-30T13:28:42.709 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.56896.log.gz 2024-03-30T13:28:42.709 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38921.log 2024-03-30T13:28:42.709 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35067.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.35067.log.gz 2024-03-30T13:28:42.709 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64281.log 2024-03-30T13:28:42.709 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.38921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65640.log 2024-03-30T13:28:42.709 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.38921.log.gz 2024-03-30T13:28:42.709 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64281.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64281.log.gz 2024-03-30T13:28:42.709 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36160.log 2024-03-30T13:28:42.709 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35952.log 2024-03-30T13:28:42.709 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65640.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.65640.log.gz 2024-03-30T13:28:42.709 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36160.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.36160.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63657.log 2024-03-30T13:28:42.709 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.709 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35952.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.35952.log.gz 2024-03-30T13:28:42.709 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46426.log 2024-03-30T13:28:42.710 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49639.log 2024-03-30T13:28:42.710 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63657.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.63657.log.gz 2024-03-30T13:28:42.710 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46426.log: 47.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.46426.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.31012.log 2024-03-30T13:28:42.710 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.710 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49639.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.49639.log.gz 2024-03-30T13:28:42.710 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56792.log 2024-03-30T13:28:42.710 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39444.log 2024-03-30T13:28:42.710 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31012.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.31012.log.gz 2024-03-30T13:28:42.710 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56792.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.56792.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36810.log 2024-03-30T13:28:42.710 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.710 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.39444.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.39444.log.gz 2024-03-30T13:28:42.710 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64203.log 2024-03-30T13:28:42.710 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65770.log 2024-03-30T13:28:42.710 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.36810.log: /var/log/ceph/ceph-client.admin.64203.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.64203.log.gz 2024-03-30T13:28:42.710 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.36810.log.gz 2024-03-30T13:28:42.711 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31028.log 2024-03-30T13:28:42.711 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45099.log 2024-03-30T13:28:42.711 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.65770.log.gz 2024-03-30T13:28:42.711 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34469.log 2024-03-30T13:28:42.711 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.31028.log: /var/log/ceph/ceph-client.admin.45099.log: 44.9% -- replaced with /var/log/ceph/ceph-client.admin.31028.log.gz 46.7% -- replaced with /var/log/ceph/ceph-client.admin.45099.log.gz 2024-03-30T13:28:42.711 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.711 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34651.log 2024-03-30T13:28:42.711 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35561.log 2024-03-30T13:28:42.711 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.34469.log.gz 2024-03-30T13:28:42.711 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.34651.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.34651.log.gz 2024-03-30T13:28:42.711 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58982.log 2024-03-30T13:28:42.711 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54187.log 2024-03-30T13:28:42.711 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35561.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.35561.log.gz 2024-03-30T13:28:42.711 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.58982.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.58982.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63943.log 2024-03-30T13:28:42.711 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.712 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54187.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54187.log.gz 2024-03-30T13:28:42.712 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33295.log 2024-03-30T13:28:42.712 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50085.log 2024-03-30T13:28:42.712 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63943.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.63943.log.gz 2024-03-30T13:28:42.712 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33295.log: 46.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52281.log 2024-03-30T13:28:42.712 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33295.log.gz 2024-03-30T13:28:42.712 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50085.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.50085.log.gz 2024-03-30T13:28:42.712 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56714.log 2024-03-30T13:28:42.712 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35874.log 2024-03-30T13:28:42.712 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52281.log: /var/log/ceph/ceph-client.admin.56714.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.52281.log.gz 2024-03-30T13:28:42.712 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.56714.log.gz 2024-03-30T13:28:42.712 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56922.log 2024-03-30T13:28:42.712 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65588.log 2024-03-30T13:28:42.712 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35874.log: /var/log/ceph/ceph-client.admin.56922.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.35874.log.gz 2024-03-30T13:28:42.712 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.56922.log.gz 2024-03-30T13:28:42.713 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39340.log 2024-03-30T13:28:42.713 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48338.log 2024-03-30T13:28:42.713 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65588.log: /var/log/ceph/ceph-client.admin.39340.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.65588.log.gz 2024-03-30T13:28:42.713 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.39340.log.gz 2024-03-30T13:28:42.713 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60580.log 2024-03-30T13:28:42.713 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62509.log 2024-03-30T13:28:42.713 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.48338.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.48338.log.gz 2024-03-30T13:28:42.713 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60580.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.60580.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63527.log 2024-03-30T13:28:42.713 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.713 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.62509.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.62509.log.gz 2024-03-30T13:28:42.713 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57546.log 2024-03-30T13:28:42.713 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40486.log 2024-03-30T13:28:42.713 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63527.log: 47.7% -- replaced with /var/log/ceph/ceph-client.admin.63527.log.gz 2024-03-30T13:28:42.713 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.57546.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.57546.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32723.log 2024-03-30T13:28:42.713 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.713 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40486.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.40486.log.gz 2024-03-30T13:28:42.714 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65848.log 2024-03-30T13:28:42.714 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35379.log 2024-03-30T13:28:42.714 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.32723.log.gz 2024-03-30T13:28:42.714 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65848.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.65848.log.gz 2024-03-30T13:28:42.721 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62405.log 2024-03-30T13:28:42.721 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37645.log 2024-03-30T13:28:42.721 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35379.log: /var/log/ceph/ceph-client.admin.62405.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.62405.log.gz 2024-03-30T13:28:42.722 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.35379.log.gz 2024-03-30T13:28:42.722 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45203.log 2024-03-30T13:28:42.722 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50917.log 2024-03-30T13:28:42.722 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.37645.log.gz 2024-03-30T13:28:42.722 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.45203.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.45203.log.gz 2024-03-30T13:28:42.722 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64782.log 2024-03-30T13:28:42.722 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63787.log 2024-03-30T13:28:42.722 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.50917.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.50917.log.gz 2024-03-30T13:28:42.722 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.64782.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.64782.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63475.log 2024-03-30T13:28:42.722 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.722 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63787.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.63787.log.gz 2024-03-30T13:28:42.722 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66108.log 2024-03-30T13:28:42.722 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40564.log 2024-03-30T13:28:42.722 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.63475.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.63475.log.gz 2024-03-30T13:28:42.723 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.66108.log: 46.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.66108.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.46322.log 2024-03-30T13:28:42.723 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.723 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.40564.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.40564.log.gz 2024-03-30T13:28:42.723 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30820.log 2024-03-30T13:28:42.723 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54083.log 2024-03-30T13:28:42.723 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.46322.log: /var/log/ceph/ceph-client.admin.30820.log: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.46322.log.gz 2024-03-30T13:28:42.723 INFO:teuthology.orchestra.run.smithi067.stderr: 47.2% -- replaced with /var/log/ceph/ceph-client.admin.30820.log.gz 2024-03-30T13:28:42.723 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37723.log 2024-03-30T13:28:42.723 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65874.log 2024-03-30T13:28:42.723 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.54083.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.54083.log.gz 2024-03-30T13:28:42.723 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37723.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.37723.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35718.log 2024-03-30T13:28:42.723 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.723 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65874.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.65874.log.gz 2024-03-30T13:28:42.723 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60190.log 2024-03-30T13:28:42.723 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56532.log 2024-03-30T13:28:42.724 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35718.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.35718.log.gz 2024-03-30T13:28:42.724 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.60190.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.60190.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37411.log 2024-03-30T13:28:42.724 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.724 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56532.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.56532.log.gz 2024-03-30T13:28:42.724 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37619.log 2024-03-30T13:28:42.724 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35926.log 2024-03-30T13:28:42.724 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37411.log: /var/log/ceph/ceph-client.admin.37619.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.37619.log.gz 2024-03-30T13:28:42.724 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.37411.log.gz 2024-03-30T13:28:42.724 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49275.log 2024-03-30T13:28:42.728 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.35926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40824.log 2024-03-30T13:28:42.728 INFO:teuthology.orchestra.run.smithi067.stderr: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.35926.log.gz 2024-03-30T13:28:42.728 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37959.log 2024-03-30T13:28:42.728 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.49275.log: /var/log/ceph/ceph-client.admin.40824.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.49275.log.gz 2024-03-30T13:28:42.728 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.40824.log.gz 2024-03-30T13:28:42.728 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52885.log 2024-03-30T13:28:42.728 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.37959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63605.log 2024-03-30T13:28:42.729 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.37959.log.gz 2024-03-30T13:28:42.729 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33269.log 2024-03-30T13:28:42.729 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.52885.log: /var/log/ceph/ceph-client.admin.63605.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.52885.log.gz 46.7% -- replaced with /var/log/ceph/ceph-client.admin.63605.log.gz 2024-03-30T13:28:42.729 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:42.729 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61703.log 2024-03-30T13:28:42.729 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.33269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65822.log 2024-03-30T13:28:42.729 INFO:teuthology.orchestra.run.smithi067.stderr: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.33269.log.gz 2024-03-30T13:28:42.729 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.61703.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.61703.log.gz 2024-03-30T13:28:42.729 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32123.log 2024-03-30T13:28:42.729 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56246.log 2024-03-30T13:28:42.729 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.65822.log: 46.7% -- replaced with /var/log/ceph/ceph-client.admin.65822.log.gz 2024-03-30T13:28:42.729 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.32123.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.32123.log.gz 2024-03-30T13:28:42.729 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.56246.log: 46.3% -- replaced with /var/log/ceph/ceph-client.admin.56246.log.gz 2024-03-30T13:28:43.310 INFO:teuthology.orchestra.run.smithi082.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-30T13:28:43.476 INFO:teuthology.orchestra.run.smithi067.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-30T13:28:43.857 INFO:teuthology.orchestra.run.smithi067.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-30T13:28:44.319 INFO:teuthology.orchestra.run.smithi067.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-30T13:28:44.806 INFO:teuthology.orchestra.run.smithi067.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-30T13:28:44.907 INFO:teuthology.orchestra.run.smithi082.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-30T13:28:45.020 INFO:teuthology.orchestra.run.smithi067.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-30T13:28:45.108 INFO:teuthology.orchestra.run.smithi082.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-30T13:28:45.189 INFO:teuthology.orchestra.run.smithi067.stderr: 92.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-30T13:28:45.191 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-03-30T13:28:45.191 INFO:teuthology.orchestra.run.smithi067.stderr:real 0m3.000s 2024-03-30T13:28:45.191 INFO:teuthology.orchestra.run.smithi067.stderr:user 0m13.039s 2024-03-30T13:28:45.191 INFO:teuthology.orchestra.run.smithi067.stderr:sys 0m0.640s 2024-03-30T13:28:45.198 INFO:teuthology.orchestra.run.smithi082.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-30T13:28:45.552 INFO:teuthology.orchestra.run.smithi082.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-30T13:28:45.554 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-03-30T13:28:45.554 INFO:teuthology.orchestra.run.smithi082.stderr:real 0m3.320s 2024-03-30T13:28:45.554 INFO:teuthology.orchestra.run.smithi082.stderr:user 0m12.743s 2024-03-30T13:28:45.554 INFO:teuthology.orchestra.run.smithi082.stderr:sys 0m0.469s 2024-03-30T13:28:45.555 INFO:tasks.ceph:Archiving logs... 2024-03-30T13:28:45.555 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630730/remote/smithi067/log 2024-03-30T13:28:45.556 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-30T13:28:47.915 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630730/remote/smithi082/log 2024-03-30T13:28:47.916 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-30T13:28:49.630 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-30T13:28:49.639 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa/tasks/rados.py", line 299, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa/tasks/rados.py", line 280, in thread run.wait(tests.values()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' 2024-03-30T13:28:49.640 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-30T13:28:49.640 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-30T13:28:49.642 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-30T13:28:49.721 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-30T13:28:49.730 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-30T13:28:49.730 DEBUG:teuthology.orchestra.run.smithi067:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T13:28:49.733 DEBUG:teuthology.orchestra.run.smithi082:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-30T13:28:49.744 INFO:teuthology.orchestra.run.smithi067.stdout: remote refid st t when poll reach delay offset jitter 2024-03-30T13:28:49.744 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================== 2024-03-30T13:28:49.744 INFO:teuthology.orchestra.run.smithi067.stdout:*hv01.front.sepi 44.4.53.2 2 u 1 64 377 0.079 -4.542 0.657 2024-03-30T13:28:49.744 INFO:teuthology.orchestra.run.smithi067.stdout:+hv02.front.sepi 63.231.80.2 3 u 55 64 377 0.083 -4.677 0.929 2024-03-30T13:28:49.744 INFO:teuthology.orchestra.run.smithi067.stdout:+hv03.front.sepi 74.6.168.72 3 u 63 64 377 0.104 -4.539 0.486 2024-03-30T13:28:49.744 INFO:teuthology.orchestra.run.smithi067.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-03-30T13:28:49.746 INFO:teuthology.orchestra.run.smithi082.stdout: remote refid st t when poll reach delay offset jitter 2024-03-30T13:28:49.746 INFO:teuthology.orchestra.run.smithi082.stdout:============================================================================== 2024-03-30T13:28:49.746 INFO:teuthology.orchestra.run.smithi082.stdout:*hv01.front.sepi 44.4.53.2 2 u 61 64 377 0.090 -3.217 1.250 2024-03-30T13:28:49.746 INFO:teuthology.orchestra.run.smithi082.stdout:+hv02.front.sepi 63.231.80.2 3 u 60 64 377 0.057 -3.747 1.152 2024-03-30T13:28:49.746 INFO:teuthology.orchestra.run.smithi082.stdout:+hv03.front.sepi 74.6.168.72 3 u 63 64 377 0.125 -2.706 1.669 2024-03-30T13:28:49.746 INFO:teuthology.orchestra.run.smithi082.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-03-30T13:28:49.747 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-30T13:28:49.773 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-30T13:28:49.774 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-30T13:28:49.815 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-30T13:28:49.861 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-30T13:28:49.886 INFO:teuthology.task.internal:Duration was 2601.518021 seconds 2024-03-30T13:28:49.887 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-30T13:28:49.914 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-30T13:28:49.914 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-30T13:28:49.917 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-30T13:28:49.954 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-30T13:28:49.954 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi067.front.sepia.ceph.com 2024-03-30T13:28:49.954 DEBUG:teuthology.orchestra.run.smithi067:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-30T13:28:50.005 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi082.front.sepia.ceph.com 2024-03-30T13:28:50.005 DEBUG:teuthology.orchestra.run.smithi082:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-30T13:28:50.015 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-30T13:28:50.016 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T13:28:50.053 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-30T13:28:50.082 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-30T13:28:50.082 DEBUG:teuthology.orchestra.run.smithi067:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-30T13:28:50.125 DEBUG:teuthology.orchestra.run.smithi082:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-30T13:28:50.202 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-30T13:28:50.212 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-30T13:28:50.213 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-30T13:28:50.225 DEBUG:teuthology.orchestra.run.smithi082:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-30T13:28:50.237 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-30T13:28:50.253 DEBUG:teuthology.orchestra.run.smithi067:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-30T13:28:50.273 DEBUG:teuthology.orchestra.run.smithi082:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-30T13:28:50.283 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = core 2024-03-30T13:28:50.291 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = core 2024-03-30T13:28:50.603 DEBUG:teuthology.orchestra.run.smithi067:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-30T13:28:50.608 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi067.front.sepia.ceph.com, flagging run as failed 2024-03-30T13:28:50.609 DEBUG:teuthology.orchestra.run.smithi082:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-30T13:28:50.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:28:50.650 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-30T13:28:50.662 INFO:teuthology.task.internal:Transferring archived files... 2024-03-30T13:28:50.663 DEBUG:teuthology.misc:Transferring archived files from smithi067:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630730/remote/smithi067 2024-03-30T13:28:50.663 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-30T13:28:52.703 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2024-03-30T13:28:52.713 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630730/remote/smithi067/coredump/1711803347.31802.core: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph_test_rados --no-omap --max-ops 4000 --objects 50 --max-in-flight 16 --size', real uid: 1000, effective uid: 1000, real gid: 1271, effective gid: 1271, execfn: '/usr/bin/ceph_test_rados', platform: 'x86_64' 2024-03-30T13:28:52.713 INFO:teuthology.task.internal: dump_program: ceph_test_rados --no-omap --max-ops 4000 --objects 50 --max-in-flight 16 --size 2024-03-30T13:28:52.714 DEBUG:teuthology.orchestra.run.smithi067:> which 'ceph_test_rados --no-omap --max-ops 4000 --objects 50 --max-in-flight 16 --size' 2024-03-30T13:28:52.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:28:52.738 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa/tasks/rados.py", line 299, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa/tasks/rados.py", line 280, in thread run.wait(tests.values()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi067 with status 1: "which 'ceph_test_rados --no-omap --max-ops 4000 --objects 50 --max-in-flight 16 --size'" 2024-03-30T13:28:52.739 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-30T13:28:52.750 INFO:teuthology.task.internal:Not uploading archives. 2024-03-30T13:28:52.751 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-30T13:28:52.787 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-30T13:28:52.787 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-30T13:28:52.790 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-30T13:28:52.794 INFO:teuthology.orchestra.run.smithi067.stdout: 658467 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 30 13:28 /home/ubuntu/cephtest 2024-03-30T13:28:52.794 INFO:teuthology.orchestra.run.smithi067.stdout: 658468 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 30 12:45 /home/ubuntu/cephtest/archive 2024-03-30T13:28:52.795 INFO:teuthology.orchestra.run.smithi067.stdout: 658470 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 30 13:28 /home/ubuntu/cephtest/archive/syslog 2024-03-30T13:28:52.795 INFO:teuthology.orchestra.run.smithi067.stdout: 658472 4 -rw-rw-rw- 1 ubuntu ubuntu 1036 Mar 30 13:28 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-03-30T13:28:52.795 INFO:teuthology.orchestra.run.smithi067.stdout: 658471 192 -rw-rw-r-- 1 ubuntu ubuntu 195378 Mar 30 13:28 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-30T13:28:52.796 INFO:teuthology.orchestra.run.smithi067.stdout: 663600 192 -rw-rw-rw- 1 ubuntu ubuntu 193090 Mar 30 13:28 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-03-30T13:28:52.796 INFO:teuthology.orchestra.run.smithi067.stdout: 658469 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 30 12:55 /home/ubuntu/cephtest/archive/coredump 2024-03-30T13:28:52.796 INFO:teuthology.orchestra.run.smithi067.stdout: 674470 17032 -rw------- 1 ubuntu ubuntu 127799296 Mar 30 12:55 /home/ubuntu/cephtest/archive/coredump/1711803347.31802.core 2024-03-30T13:28:52.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-30T13:28:52.797 INFO:teuthology.orchestra.run.smithi067.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-30T13:28:52.797 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa/tasks/rados.py", line 299, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph-c_5dc2ea3016dcf8ef9e88b80203119d1d5e6f94cc/qa/tasks/rados.py", line 280, in thread run.wait(tests.values()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi067 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-03-30T13:28:52.797 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-30T13:28:52.860 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-30T13:28:52.900 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0' 2024-03-30T13:28:52.900 INFO:teuthology.nuke:Checking targets against current locks 2024-03-30T13:28:52.904 INFO:teuthology.orchestra.run.smithi082.stdout: 658624 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 30 13:28 /home/ubuntu/cephtest 2024-03-30T13:28:52.905 INFO:teuthology.orchestra.run.smithi082.stdout: 658625 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 30 13:28 /home/ubuntu/cephtest/archive 2024-03-30T13:28:52.905 INFO:teuthology.orchestra.run.smithi082.stdout: 658627 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 30 13:28 /home/ubuntu/cephtest/archive/syslog 2024-03-30T13:28:52.905 INFO:teuthology.orchestra.run.smithi082.stdout: 658629 4 -rw-rw-rw- 1 ubuntu ubuntu 833 Mar 30 13:28 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-03-30T13:28:52.905 INFO:teuthology.orchestra.run.smithi082.stdout: 658628 176 -rw-rw-r-- 1 ubuntu ubuntu 178794 Mar 30 13:28 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-30T13:28:52.905 INFO:teuthology.orchestra.run.smithi082.stdout: 663757 164 -rw-rw-rw- 1 ubuntu ubuntu 166734 Mar 30 13:28 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-03-30T13:28:52.906 INFO:teuthology.orchestra.run.smithi082.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-30T13:28:52.929 DEBUG:teuthology.nuke:shortname: smithi067 2024-03-30T13:28:52.929 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T13:28:52.955 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630730', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 12:36:52.783281', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHc368b/ld8CXeRgZD2wZKDabNI1eXfuXEMNsxKrM0ylBNhZkR2Ojg21ljXQq705FBt6xWiWXgxuXG2KNvfnxwY='} 2024-03-30T13:28:52.958 DEBUG:teuthology.nuke:shortname: smithi082 2024-03-30T13:28:52.958 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-30T13:28:52.981 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-30_00:41:04-rados-wip-ceph_test_rados-partial-reads-distro-default-smithi/7630730', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-30 12:36:52.781685', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOSfXZVLelftUITjlRAgf/G/9OTlf8HfMEZz+qIw9jTkkEqgB1v5m3QDgDoN4xfB9xrvptDpktppp2QW52tuIEI='} 2024-03-30T13:28:53.004 INFO:teuthology.orchestra.console.smithi067:Power off 2024-03-30T13:28:53.004 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T13:28:53.028 DEBUG:teuthology.orchestra.console.smithi067:power off output: Chassis Power Control: Down/Off 2024-03-30T13:28:53.029 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T13:28:53.045 INFO:teuthology.orchestra.console.smithi082:Power off 2024-03-30T13:28:53.045 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-30T13:28:53.055 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-03-30T13:28:53.071 DEBUG:teuthology.orchestra.console.smithi082:power off output: Chassis Power Control: Down/Off 2024-03-30T13:28:53.072 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T13:28:53.095 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2024-03-30T13:28:57.057 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T13:28:57.096 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T13:28:57.170 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-03-30T13:28:57.207 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2024-03-30T13:29:01.172 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T13:29:01.209 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T13:29:01.283 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is off 2024-03-30T13:29:01.319 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2024-03-30T13:29:01.384 INFO:teuthology.orchestra.console.smithi067:Power off completed 2024-03-30T13:29:05.321 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-30T13:29:05.432 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is off 2024-03-30T13:29:05.533 INFO:teuthology.orchestra.console.smithi082:Power off completed 2024-03-30T13:29:05.638 INFO:teuthology.run:Summary data: description: rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-snaps-few-objects-overwrites} duration: 2601.5180213451385 failure_reason: 'Command crashed: ''CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0''' flavor: default owner: scheduled_yuriw@teuthology status: fail success: false 2024-03-30T13:29:05.639 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-30T13:29:05.729 INFO:teuthology.run:FAIL